Netstat not showing closing connections from Server Status (under DOS attack)

[stars]

I am suffering form DOS attack on apache. My apache reaches max_clients (now set to 1800) and stops responding. Case would be simple because CSF/LDF has feature of tracking connections on port 80 and block offender after reaching CT_LIMIT (set to 300) number of connections but it seems it doesnt work in my case.

All connections are from one IP, all connections has status "Closing connection" and netstat doesnt show any of these connections at all.

How can I track these connections? I wrote a script to block IP and restart httpd but if netstat doesnt show these connections I cannot use it.

This is server status output:

13.8 requests/sec - 103.4 kB/second - 7.5 kB/request
1251 requests currently being processed, 10 idle workers

CCCCCC_CCCCCCCCCCC_CCCCCCCCCCCCCCCCCCCCCCCCCCCCCWCCCC.CCCCCCC.CC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC_CWCCCCCCCCCCC
CCCCCCCCC_CCCCCCCCCCCCCCCCK_CCCWCCCCCCCCCC.CCCCCCCCCCCCCCCCCCCCC
C_CCCCCCCCCCCCCCCCC.CCCCCCCCC.CCCCCCCCCCCCCC.CCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC.CCCCCC_CCCCCCCKCCC.CCC_CCCCCC
CCCCCCCCCC_CCCCCC.CCCCCKCCCCCCCCCCCCC.CCCC.CCCCC.CCCC.CCCCCCCCCC
.CCC.C.CCCCCCCCC...CCC.CCCCCCCCC.CCC.CCCCCCCCCC..CCCCCCCCCCCCCCC
CCCCCCCCCCCCC.CCCCCCCCCCCCCCC.CCCCCCCCCCCCCCCCCCCCCCC..CCCCCCCCC
CCCC.CC.CC.CCCC.CCCCCCCCCCC.CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC.
CCCCCCCCCCCCCCC.CCCCCCCCCCCC.C.CCCCCCCCCCCC..CCCCCCCCC.CCCCCCCCC
CCCCCC.C.CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC.CCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCC.CCCCCCC.CCCCCCCCCCCCC.CCCCCCCCCCCCCCCC.CCCCCC
CCCCCCCCCCCCCCCCCCCCC.C..CCCCCCCCCCCCCCCCCCCCCC.CCCCCCCCCCCC.CCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC_CCCCCCCCCC.CC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC.CCCC.CCCCCCCCCCCCCCCCCCCC.CCCCCCC
CCCCCCCCCCCCCCCCC..CCCCC.CCCCCCCCCCCCCCCCCCCCCCCCCCCCC.CCC.CCCCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC.CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
CC.CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCC.CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC.CCCCCCCCCCCCC
CCCCCCCCCC.CCCCCCCCCCCCCCCCCCCCCCCCCCCC.CCCCCCCCCCCCCCCCCCCCCCCC
CC.CCCC.CCCCCCCCCCCCCCC.CCCCCCCCC.C.C..C.CCC.C...CC...C..CC..CC.
......CC..C......C..............................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
........

netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
34 ESTABLISHED
      3 FIN_WAIT1
     15 FIN_WAIT2
      1 LAST_ACK
     26 LISTEN
    382 TIME_WAIT

Is there a way to track these connections or close them faster?

 

[sysdev]

First of all.. if it's just one ip, block it in csf or whatever firewall you're using.
Next, check your current timeout time ( cat /proc/sys/net/ipv4/tcp_fin_timeout ) and lower it ( e.g. echo 20 > /proc/sys/net/ipv4/tcp_fin_timeout ). That will close them a lot faster.
netstat -an|grep TIME_WAIT | awk '{print $5}' shows the ip-addresses having connections in a time_wait status. Count them, uniq them etc. and block the top 5 every minute if the count is > 100.
Remember to unblock them again if your serverload goes down because you might be blocking proxies used on e.g. universities.

 

[stars]

Thanks for reply. If I block one IP, in a day or two another one will come up, so I want to automate it somehow or write some alert script.

My current timeout is 60s which is ok, I guess.

netstat command as I wrote before doesn't give me same results as server-status. All I need to know is how many requests currently being processed. How can I get this variable, beside server-status page?