Help: Too Many Connections Error -> System Stop

[nicsopon]

The system has too may connections and stop for along time. I've increased

I'm using CentOS5.0 and DA CustomApache(apache1.3 and php 4.4.7)

MaxClients 3000 (from 450)

The problem is still existed.


From vi /var/log/httpd/error_log get a lot of (4 sections)

Dec 8 12:24:35 2007] [error] [client 66.249.72.138] File does not exist: /var/www/html/webboard/index.php
[Sat Dec 8 12:24:39 2007] [error] [client 38.114.104.119] File does not exist: /var/www/html/webboard/index.php
[Sat Dec 8 12:24:41 2007] [error] [client 66.249.72.138] File does not exist: /var/www/html/webboard/index.php
[Sat Dec 8 12:24:56 2007] [error] [client 38.114.104.119] File does not exist: /var/www/html/webboard/index.php
[Sat Dec 8 12:24:56 2007] [error] [client 66.249.72.138] File does not exist: /var/www/html/robots.txt
[Sat Dec 8 12:25:22 2007] [error] [client 66.249.72.138] File does not exist: /var/www/html/product_reviews.php
[Sat Dec 8 12:25:43 2007] [error] [client 66.249.72.138] File does not exist: /var/www/html/webboard/index.php
[Sat Dec 8 12:26:08 2007] [error] [client 66.249.72.138] File does not exist: /var/www/html/gallery/main.php
[Sat Dec 8 12:26:25 2007] [error] [client 66.249.72.138] File does not exist: /var/www/html/gallery/main.php
and more ....

[Sat Dec 8 12:42:25 2007] [notice] child pid 27696 exit signal Segmentation fault (11)
[Sat Dec 8 12:42:25 2007] [notice] child pid 27697 exit signal Segmentation fault (11)
[Sat Dec 8 12:42:25 2007] [notice] child pid 27698 exit signal Segmentation fault (11)
[Sat Dec 8 12:42:25 2007] [notice] child pid 27699 exit signal Segmentation fault (11)
[Sat Dec 8 12:42:25 2007] [notice] child pid 27700 exit signal Segmentation fault (11)
[Sat Dec 8 12:42:25 2007] [notice] child pid 27701 exit signal Segmentation fault (11)
[Sat Dec 8 12:42:25 2007] [notice] child pid 27702 exit signal Segmentation fault (11)
[Sat Dec 8 12:42:25 2007] [notice] child pid 27704 exit signal Segmentation fault (11)
[Sat Dec 8 12:42:26 2007] [notice] child pid 27703 exit signal Segmentation fault (11)
[Sat Dec 8 12:42:26 2007] [notice] child pid 27705 exit signal Segmentation fault (11)
and more ...


*** glibc detected *** /usr/sbin/httpd: double free or corruption (!prev): 0x00000000128e2d90 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2aaaab3f0890]
/lib64/libc.so.6(cfree+0x8c)[0x2aaaab3f3fac]
/usr/lib/apache/libphp4.so(_efree+0xc7)[0x2aaaaf88db17]
/usr/lib/apache/libphp4.so(php_end_ob_buffer+0x5d8)[0x2aaaaf881c08]
/usr/lib/apache/libphp4.so(php_end_ob_buffers+0x29)[0x2aaaaf882059]
/usr/lib/apache/libphp4.so(php_request_shutdown+0x3ab)[0x2aaaaf86f10b]
/usr/lib/apache/libphp4.so[0x2aaaaf8bb024]
/usr/sbin/httpd[0x40e045]
/usr/sbin/httpd(ap_clear_pool+0x3a)[0x40c0fe]
/usr/sbin/httpd(ap_destroy_pool+0x1a)[0x40c185]
/usr/sbin/httpd(ap_clear_pool+0x20)[0x40c0e4]
/usr/sbin/httpd[0x4201f3]
/usr/sbin/httpd[0x42099d]
/usr/sbin/httpd[0x420dc8]
/usr/sbin/httpd[0x421455]
/usr/sbin/httpd(main+0x3a8)[0x421aad]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2aaaab39f8a4]
/usr/sbin/httpd[0x40a049]
======= Memory map: ========
[Sat Dec 8 21:56:38 2007] [notice] child pid 31940 exit signal Aborted (6)
[Sat Dec 8 21:57:46 2007] [error] Bad pid (31940) in scoreboard slot 117
[Sat Dec 8 21:57:46 2007] [error] Bad pid (335) in scoreboard slot 177
[Sat Dec 8 21:57:46 2007] [error] Bad pid (402) in scoreboard slot 179
[Sat Dec 8 21:57:46 2007] [error] Bad pid (415) in scoreboard slot 183
[Sat Dec 8 21:57:46 2007] [error] Bad pid (650) in scoreboard slot 184
[Sat Dec 8 21:57:46 2007] [error] Bad pid (651) in scoreboard slot 185
[Sat Dec 8 21:57:46 2007] [error] Bad pid (654) in scoreboard slot 187
[Sat Dec 8 21:57:46 2007] [error] Bad pid (680) in scoreboard slot 190
[Sat Dec 8 21:57:46 2007] [error] Bad pid (681) in scoreboard slot 191

[Wed Dec 12 13:26:34 2007] [warn] module perl_module is already loaded, skipping
[Wed Dec 12 13:26:34 2007] [notice] Apache/1.3.39 (Unix) mod_ssl/2.8.30 OpenSSL/0.9.8b PHP/4.4.7 mod_perl/1.29 FrontPage/5.0.2.2510 configured -- resuming normal operations
[Wed Dec 12 13:26:34 2007] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Dec 12 13:26:34 2007] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Wed Dec 12 14:22:25 2007] [error] [client 203.172.186.10] request failed: erroneous characters after protocol string: GET /webboard/\\x02%CC%A3:?LZd?M?\\xd3\\xbaQ?]?-/N\\x14j\\x05>?\\xea\\xb4\\xb1?\\xd0\\x8as?\\x05\\xc9\\x91v??V?\\xd0\\xb5?2?\\xe2\\xaa\\xba\\x05\\x02?\\xd1\\xbd]N\\xd5\\x9a?\\x10?S\\x15?.\\xe9\\x9d\\xb7P\\x14\\x10E5:?VA\\x03W??f??)?\\xd4\\xaf]\\xd7\\x85\\xe9\\x8e\\xac?]VF\\xcd\\x90\\x10?O?\\xea\\x93\\xbb\\x1c?R\\x1b?\\x1bwN\\xeb\\xbb\\x80A\\x14\\x13??\\xe8\\x8d\\xb4?(\\x13?m\\x04\\xec\\xa2\\xb6V\\xdb\\xa0?Wt]*hh\\xcf\\xadZ??{;?\\v??%20?@??:\\xeb\\x87\\xaa?) HTTP/1.0
[Wed Dec 12 16:43:49 2007] [error] Bad pid (9701) in scoreboard slot 139
[Wed Dec 12 16:44:43 2007] [warn] module perl_module is already loaded, skipping
[Wed Dec 12 16:44:43 2007] [notice] Apache/1.3.39 (Unix) mod_ssl/2.8.30 OpenSSL/0.9.8b PHP/4.4.7 mod_perl/1.29 FrontPage/5.0.2.2510 configured -- resuming normal operations
[Wed Dec 12 16:44:43 2007] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Dec 12 16:44:43 2007] [notice] Accept mutex: sysvsem (Default: sysvsem)

It's really over my power to find the way to fix the problem. Please give me some advice what I should do to reduce these or where the problems come from.

Thanks

 

[dwm]

Are you sure you are not under attack? Ask your ISP for this. It looks like alot of clients are requesting urls for exploiting / hitting your server.

 

[andyreed]

The system has too may connections and stop for along time. I've increased

I'm using CentOS5.0 and DA CustomApache(apache1.3 and php 4.4.7)

MaxClients 3000 (from 450)

The problem is still existed.

I agree with what dwm has said. It is likely that your Web server is under some sort of Internet attack. SSH to the server and run this command and post the results here:
/bin/netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

Just in case you need to learn about Internet attacks, go to:
http://www.servertune.com/kbase/entry/64/
and
http://www.servertune.com/kbase/entry/67/

 

[shanky]

Result as requested

The result of

/bin/netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

[root@ns1 ~]# /bin/netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
18 203.155.231.57
10 222.123.155.122
9 203.170.228.168
7 125.24.243.235
7 124.120.229.15
6 61.7.131.108
6 202.28.179.3
6 125.26.235.199
5 58.9.222.42
4 61.114.254.18
4 58.136.75.26
4 124.120.136.8
4 116.58.227.251
3 61.208.142.246
3 58.9.146.55
3 124.121.59.69
2 58.9.193.100
2 58.9.190.71
2 58.9.101.24
2 222.123.227.213
2 210.203.186.57
2
1 servers)
1 Address
1 82.181.239.115
1 77.249.23.186
1 75.125.244.51
1 66.249.72.138
1 62.226.188.156
1 61.7.166.72
1 58.9.79.210
1 58.8.99.105
1 58.8.86.155
1 58.8.133.28
1 58.8.105.35
1 58.8.103.131
1 58.64.39.28
1 203.113.35.12
1 134.106.106.22
1 125.26.114.27
1 125.25.219.154
1 124.121.26.127
1 124.120.17.5
[root@ns1 ~]#

Some more idea for the result please.
Thanks

 

[shanky]

One High Traffice Site

Some More Info.

My Server has 2 GB of Ram.

I have one customer wiht high active connection about minimum 200 connections and up to >500 at the same time. I see from the "Online Info of the SMF Boards"

This may be usefull info for you

Thanks

 

[andyreed]

My Server has 2 GB of Ram.

I have one customer wiht high active connection about minimum 200 connections and up to >500 at the same time. I see from the "Online Info of the SMF Boards"

To see the imapct of this client on your server CPU load, suspend him or disable his account for few minutes. From the results posted, you are not under Syn Flooding attack. However, it is really hard to say without looking into your server. Internet attacks are difficult to identify from one or two postings of results. Contact your host for help, or you can seek professional help.