too many concurrent SMTP connections Error 421

SupermanInNY

Verified User
Joined
Sep 28, 2004
Messages
419
I'm getting hit with too many concurrent SMTP connections.
That's not something that I've ever seen, so I'm suspecting this is some kind of an attack of some sort.

29327 ? Ss 0:00 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29355 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29356 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29359 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29360 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29361 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29362 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29364 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29365 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29366 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29367 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29368 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29369 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29370 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29371 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29373 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29375 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29377 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29379 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29380 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29381 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29385 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29386 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29387 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29388 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29389 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29390 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
29394 ? S 0:00 \_ /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid


and my mainlog gets many of these:

2007-10-02 01:00:38 Connection from [83.10.118.226] refused: too many connections
2007-10-02 01:00:38 Connection from [71.181.161.186] refused: too many connections
2007-10-02 01:00:38 Connection from [75.132.15.88] refused: too many connections
2007-10-02 01:00:38 Connection from [86.151.160.215] refused: too many connections
2007-10-02 01:00:38 Connection from [81.129.119.104] refused: too many connections
2007-10-02 01:00:38 Connection from [75.80.4.63] refused: too many connections
2007-10-02 01:00:38 Connection from [82.31.18.228] refused: too many connections
2007-10-02 01:00:38 Connection from [65.89.103.130] refused: too many connections
2007-10-02 01:00:38 Connection from [81.129.119.104] refused: too many connections
2007-10-02 01:00:38 Connection from [207.255.101.223] refused: too many connections
2007-10-02 01:00:38 Connection from [200.5.198.201] refused: too many connections
2007-10-02 01:00:38 Connection from [201.92.134.79] refused: too many connections
2007-10-02 01:00:39 Connection from [212.183.54.128] refused: too many connections
2007-10-02 01:00:39 Connection from [75.130.79.88] refused: too many connections
2007-10-02 01:00:39 Connection from [201.83.21.152] refused: too many connections
2007-10-02 01:00:39 Connection from [213.144.122.230] refused: too many connections
2007-10-02 01:00:39 Connection from [201.3.14.44] refused: too many connections
2007-10-02 01:00:39 Connection from [151.41.77.252] refused: too many connections
2007-10-02 01:00:39 Connection from [189.136.203.32] refused: too many connections
2007-10-02 01:00:39 Connection from [201.79.214.184] refused: too many connections
2007-10-02 01:00:39 Connection from [190.49.208.153] refused: too many connections
2007-10-02 01:00:40 Connection from [201.78.219.231] refused: too many connections
2007-10-02 01:00:40 Connection from [62.57.91.80] refused: too many connections
2007-10-02 01:00:40 Connection from [90.20.187.232] refused: too many connections
2007-10-02 01:00:40 Connection from [189.136.203.32] refused: too many connections
2007-10-02 01:00:40 Connection from [189.136.203.32] refused: too many connections
2007-10-02 01:00:40 Connection from [189.136.203.32] refused: too many connections
2007-10-02 01:00:40 Connection from [201.20.64.101] refused: too many connections
2007-10-02 01:00:40 Connection from [24.88.90.224] refused: too many connections
2007-10-02 01:00:40 Connection from [201.8.157.102] refused: too many connections
2007-10-02 01:00:40 Connection from [60.38.37.114] refused: too many connections
2007-10-02 01:00:40 Connection from [80.228.198.89] refused: too many connections
2007-10-02 01:00:40 Connection from [86.69.46.107] refused: too many connections
2007-10-02 01:00:40 Connection from [190.142.49.92] refused: too many connections
2007-10-02 01:00:44 Connection from [200.86.184.112] refused: too many connections
2007-10-02 01:00:44 Connection from [189.24.101.249] refused: too many connections
2007-10-02 01:00:44 Connection from [24.232.87.92] refused: too many connections
2007-10-02 01:00:45 Connection from [90.188.197.38] refused: too many connections
2007-10-02 01:00:46 Connection from [190.86.84.108] refused: too many connections
2007-10-02 01:00:46 Connection from [201.78.126.254] refused: too many connections
2007-10-02 01:00:47 Connection from [86.139.48.252] refused: too many connections
2007-10-02 01:00:47 Connection from [68.179.17.185] refused: too many connections
2007-10-02 01:00:47 Connection from [200.214.157.3] refused: too many connections

Any pointers?

-Alon
 
More information:

After I do:

service exim restart

It gets loaded within a 30 seconds.

smtp_receive_timeout = 5m
smtp_accept_max = 100

I tried to change the smtp_accept_max to 150, it got loaded few seconds later. So obviously that is not the problem.
Also,. since this all happens in less than a minute from a restart, the timeout of 5m is of no significance as the server is loaded much less than the timeout.

Is there a Maximum number of concurrent TCP connections to the SMTP server from one IP address?

This is a method of protection against DoS attacks (Denial of Service — too many concurrent connections overload the system and no other users can connect to the server).

I have spamd running in the background.

29374 ? Ss 0:00 /usr/bin/spamd -d -c -m 5
29477 ? S 0:01 \_ spamd child
29478 ? S 0:00 \_ spamd child


/var/log/exim # netstat -ano | grep "25"
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN keepalive (0.00/0/0)
tcp 0 0 111.222.333.444:25 66.233.48.215:4649 SYN_RECV on (0.43/1/0)
tcp 0 0 111.222.333.444:80 80.230.255.60:24715 ESTABLISHED keepalive (1394.88/0/0)
tcp 1 0 111.222.333.444:25 86.139.244.61:2031 CLOSE_WAIT keepalive (1126.85/0/0)
tcp 0 0 111.222.333.444:25 88.227.255.50:4483 ESTABLISHED keepalive (1381.82/0/0)
tcp 0 56 111.222.333.444:25 71.181.161.186:63228 LAST_ACK on (0.81/4/0)
tcp 0 14 111.222.333.444:25 71.181.161.186:61691 ESTABLISHED on (9.06/7/0)
tcp 0 56 111.222.333.444:25 75.168.18.132:3712 LAST_ACK on (7.17/7/0)
tcp 0 0 111.222.333.444:25 200.167.22.182:2983 TIME_WAIT timewait (45.57/0/0)
tcp 0 0 111.222.333.444:25 81.192.181.231:3123 TIME_WAIT timewait (41.96/0/0)
tcp 0 77 111.222.333.444:25 88.233.40.112:2028 FIN_WAIT1 on (31.26/9/0)
tcp 0 0 111.222.333.444:25 201.15.200.160:61379 FIN_WAIT2 keepalive (1105.60/0/0)
tcp 0 0 111.222.333.444:25 202.155.224.58:36122 TIME_WAIT timewait (10.22/0/0)
tcp 0 0 111.222.333.444:25 82.74.17.87:65365 TIME_WAIT timewait (19.21/0/0)
tcp 0 0 111.222.333.444:110 85.250.114.222:3839 TIME_WAIT timewait (25.28/0/0)
tcp 0 0 111.222.333.444:143 84.228.22.78:21225 ESTABLISHED keepalive (220.65/0/0)
tcp 0 0 111.222.333.444:80 192.115.62.166:25152 FIN_WAIT2 keepalive (443.26/0/0)
tcp 0 0 111.222.333.444:80 203.11.225.5:36029 FIN_WAIT2 keepalive (1267.35/0/0)
tcp 0 0 111.222.333.444:25 200.127.46.10:51977 TIME_WAIT timewait (41.66/0/0)
tcp 0 0 111.222.333.444:80 192.115.62.166:25413 TIME_WAIT timewait (19.92/0/0)
udp 0 0 111.222.333.444:54425 192.115.106.35:53 ESTABLISHED off (0.00/0/0)

Anyone?

-Alon.
 
Last edited:
Alon,

Long log file excerpts are wasted on me; I won't take the time to look at them. I hope you can pare your log extracts down to a few lines that show a problem.

You appear to be undergoing a DOS attack.

You wrote:
Is there a Maximum number of concurrent TCP connections to the SMTP server from one IP address?
Googling for exim max concurrent connections from IP found this as the third entry; a quote from the exim book by Dr Philip Hazel:
If smtp_accept_max_per_host is set, the list of current connections is scanned to find out how many are from the incoming IP address. If the new connection would cause the limit to be exceeded, the connection is rejected with the error:

421 Too many concurrent SMTP connections from one IOP address; please try again later.

There is no default limit on the number of connections from a single host.
Jeff
 
Alon,

Long log file excerpts are wasted on me; I won't take the time to look at them. I hope you can pare your log extracts down to a few lines that show a problem.

You appear to be undergoing a DOS attack.

You wrote:

Googling for exim max concurrent connections from IP found this as the third entry; a quote from the exim book by Dr Philip Hazel:

Jeff


Hi Jeff,

My bad on the long logs.
I cleaned it up so the next guy reading this won't see long logs.
I believe this is not a DOS attack, but rather a DDoS attack due to the magnitude of numbers of the diff IPs.

My understanding that to combat this DDoS attack I have several ways to go:

Option 1:
=======
Move the port of 25 to port 587 or some other port.

daemon_smtp_ports = 25 : 587

Does that Move the port or just replicates it to answer / listen as well?

Option 2:
=======
Increase the value
smtp_accept_max = 200 (default 100)
The risk here is running out of memory eventually.

Option 3:
=======
apf -d unlikely IP ranges that would use the SMTP port (Brazil seems to be running a good amount of the traffic).
I like this idea, but not sure of the correct syntax.

tcp:in:d=25:s=200.0.0.0/7

Is that what put as a line in the /etc/apf/deny_hosts.rules ?
Just this line as is and then run:

apf -r

Will this prevent them from killing my port 25, yet still allow them to surf to the websites?


Option 4:
=======
Get re-aquinted with the local religious traditions and apply known rituals to appeal to a higher power?

Option 5:
=======
<fill in your ideas here>

-Alon.
 
praying to higher power may lead to someone posting a fix for you :)

Is it possible to determine who's domain is under attack? If so, change TTL 7200 or less then change mail A to 127.0.0.1.

It could take to the maximum of the current TTL for the attack to stop. Once the attack stops change A record back to normal if the attack resumes change it back to localhost.

Bear in mind that the domain will not receive any email while it set to localhost, but it is better than bringing down your server and/or internet connection.

LAST resort: change the IP of domain and/or server.
 
I cleaned it up so the next guy reading this won't see long logs.
Gut. Danke.
I believe this is not a DOS attack, but rather a DDoS attack due to the magnitude of numbers of the diff IPs.
I would have too, if I'd looked at the logs. But of course a DDOS is a DOS attack; just a specialized kind.
Move the port of 25 to port 587 or some other port.

daemon_smtp_ports = 25 : 587

Does that Move the port or just replicates it to answer / listen as well?
That line should already be in exim.conf. It means it listens on both ports. Other code further down in the file limits that port to be used only by authenticated senders.

Of course if you move exim off port 25 completely you'll solve the problem, but it would be easier to just unplug the server; the effect would be roughly the same ;) . If you block port 25, then no one can send email to the server.
Increase the value
smtp_accept_max = 200 (default 100)
The risk here is running out of memory eventually.
The only way this makes sense to me is if you do it on a standalone server, just to handle the email, and heavily filter email coming in before relaying it to the real mailserver.
apf -d unlikely IP ranges that would use the SMTP port (Brazil seems to be running a good amount of the traffic).
I like this idea, but not sure of the correct syntax.
I'm not either.
Will this prevent them from killing my port 25, yet still allow them to surf to the websites?
You can block only certain ports.
Get re-aquinted with the local religious traditions and apply known rituals to appeal to a higher power?
See my private reply :) .
<fill in your ideas here>
If the attack is coming to one domain, figure out which domain is attracting the email. Let the client know it's happening, and that you need to shut him down. Then shut him down. More below, in response to bclark94.
bclark94 said:
Is it possible to determine who's domain is under attack? If so, change TTL 7200 or less then change mail A to 127.0.0.1.
Heck, change it to 600. Most DDOS attacks are to the A record of the domain as well as to the MX record. I say shut down the domain and change the A record for the domain name to 127.0.0.1, and the A record for mail to 127.0.0.1. It probably won't help, because they're probably hardwired the IP# by now, but you can try it.
It could take to the maximum of the current TTL for the attack to stop. Once the attack stops change A record back to normal if the attack resumes change it back to localhost.
We kept the mx for one of our domains to 127.0.0.1 for two years without slowing down the spam in the slightest. And of course if you set mx to 127.0.0.1 be prepared for some flaming from some admins who don't agree with the strategy. It happened to us.
Bear in mind that the domain will not receive any email while it set to localhost, but it is better than bringing down your server and/or internet connection.
That's why I recommend shutting down the user if it can be identified.
LAST resort: change the IP of domain and/or server.
If the DDOS doesn't stop on it's own, this is going to soon be the only resort with a possibility of working. And only if you delete the site permanently first.

Jeff
 
Gut. Danke.

I would have too, if I'd looked at the logs. But of course a DDOS is a DOS attack; just a specialized kind.

That line should already be in exim.conf. It means it listens on both ports. Other code further down in the file limits that port to be used only by authenticated senders.

Of course if you move exim off port 25 completely you'll solve the problem, but it would be easier to just unplug the server; the effect would be roughly the same ;) . If you block port 25, then no one can send email to the server.

The only way this makes sense to me is if you do it on a standalone server, just to handle the email, and heavily filter email coming in before relaying it to the real mailserver.

I'm not either.

You can block only certain ports.

See my private reply :) .

If the attack is coming to one domain, figure out which domain is attracting the email. Let the client know it's happening, and that you need to shut him down. Then shut him down. More below, in response to bclark94.

Heck, change it to 600. Most DDOS attacks are to the A record of the domain as well as to the MX record. I say shut down the domain and change the A record for the domain name to 127.0.0.1, and the A record for mail to 127.0.0.1. It probably won't help, because they're probably hardwired the IP# by now, but you can try it.

We kept the mx for one of our domains to 127.0.0.1 for two years without slowing down the spam in the slightest. And of course if you set mx to 127.0.0.1 be prepared for some flaming from some admins who don't agree with the strategy. It happened to us.

That's why I recommend shutting down the user if it can be identified.

If the DDOS doesn't stop on it's own, this is going to soon be the only resort with a possibility of working. And only if you delete the site permanently first.

Jeff


After a long night of no sleep, I was able to battle some of the attack.

First, the attack wasn't on a specific domain that I was able to identify. Heck.. that would be just way too easy. I'd kill that domain in a heartbeat. But, the attack was just doing "Connecting to IP:25", so it was not something that I could have done.

I did add massive blocks to Brazil, Columbia, Chille and it seem to reduce the amount of connection to some level, so I had some success of doing this.

Here is what I put into the deny_host:

###### ALON ATTACK
# added 200.0.0.0/7 on 10/02/07 03:13:34
tcp:in:d=25:s=200.0.0.0/24
# added 201.0.0.0/7 on 10/02/07 03:13:57
tcp:in:d=25:s=201.0.0.0/24
# added 190.0.0.0/7 on 10/02/07 03:14:06
tcp:in:d=25:s=190.0.0.0/24
# added 189.0.0.0/7 on 10/02/07 03:14:14
tcp:in:d=25:s=189.0.0.0/24
# added 87.0.0.0/24 on 10/02/07 09:05:38
tcp:in:d=25:s=87.0.0.0/24
# ALON END ATTACK

This is brutal block, but it did something. You can of course add other blocks, I did those at first because when I parsed the logs, those stood out as obvious. I don't have clients in that part of the world. At least not clients who needed to send emails out from there at that particular time.
This block is now commented out, I'm just keeping it for future reference.

Last, the attack stopped on 09:00AM the next morning and life started to get back to normal.

Jeff,. we had the higher power discussion before,. you seem to know much more then what you wrote in the PM :)))))).
Perhaps it helped? :).

Thanks,

-Alon.
 
how can we witch domain is under attack?
my server is under ddos attack.it start on httpd and now on exim.
first i read your iadia and so change my port to 251
but its not work and my exim process is :

exim (pid 8663 8783 8799 8929 9014 9035 9060 9165 9584 9588 9646 9812 9822 9880 9902 9913 9915 9944 9967 9999 10112 10115 10116 10145 10174 10300 10310 10357 10379 10425 10501 10512 10550 10574 10578 10608 10610 10628 10651 10654 10761 10770 10774 10775 10814 10822 10856 10870 10894 10896 10899 10930 10936 10968 10991 10997 11274 11278 11285 11329 11331 11348 11387 11418 11421 11460 11475 11523 11529 11548 11563 11572 11615 11631 11635 11686 11715 11738 11767 11792 11805 11861 11867 11897 11909 11940 11946 11963 11971 11998 12023 12055 12074 12086 12103 12127 12159 12168 12196 12223 12235 12239 12243 12279 12282 12288 12314 12323 12391 12412 12433 12437 12458 12466 12487 12497 12516 12517 12521 12522 12526 12529 12534 12537 12538 12543 12548 12549 12558 12560 12561 12562 12563 12568 12570 12575 12582 12603 12606 12608 12630 12632 12680 12688 12694 12703 12724 12734 12735 12737 12786 12788 12819 12831 12852 12854 12858 12879 12888 12905 12929 12930 12936 12939 12963 12968 12970 12982 12984 12986 13002 13004 13015 13018 13025 13031 13044 13045 13054 13056 13058 13063 13065 13067 13074 13095 13108 13110 13120 13122 13125 13129 13130 13135 13150 13151 13154 13156 13163 13177 13188 13189 13191 13212 13237 13239 13260 13268 13273 13280 13281 13282 13283 13312 13317 13338 13340 13341 13347 13370 13371 13378 13455 13509 13556 13557 13577 13588 13595 13597 13598 13603 13610 13630 13662 13664 13670 13673 13674 13682 13683 13684 13685 13709 13711 13714 13729 13730 13755 13756 13757 13758 13759 13760 13763 13767 13768 13769 13770 13774 13776 13821 13824 13841 13842 13843 13867 13873 13877 13884 13886 13888 13922 13923 13939 13942 13943 13945 13948 13950 13952 13953 13956 13960 13962 13963 13965 13966 13969 13971 13972 13974 13978 13980 14007 14024 14027 14054 14055 14057 14060 14062 14093 14094 14095 14096 14115 14131 14141 14142 14144 14154 14155 14156 14157 14158 14159 14161 14163 14191 14235 14247 14261 14273 14388 14404 14414 14432 14468 14470 14496 14498 14504 14505 14506 14553 14555 14579 14583 14584 14586 14625 14654

help me.
 
Im sure iptables can limit concurrent connections for each ip.
 
fix

I got hit with something similar... Heaps of connection attempts form about 10000 sources...

Example Log:
linweb01:/var/log/exim# tail rejectlog
2011-11-12 11:15:44 H=corp-190-110-197-210-uio.puntonet.ec [190.110.197.210] F=<[email protected]> rejected RCPT <[email protected]>: authentication required, relaying mail not allowed
2011-11-12 11:15:44 H=c9518719.virtua.com.br [201.81.135.25] F=<[email protected]> rejected RCPT <[email protected]>: authentication required, relaying mail not allowed

Here's a iptables fix for anyone thats been rejected 10 times in the rejectlog

linweb01:/var/log/exim# cat countdenied.sh
#!/usr/bin/perl

my %hash;

open(FILE,"rejectlog");
@DATA=<FILE>;
close(FILE);

foreach my $line (@DATA)
{
chomp($line);
$line =~ /\) \[(.*?)\]/;
$hash{$1}++;
}

foreach my $ipaddr (sort{$b <=> $a} keys %hash)
{
if($hash{$ipaddr}>=10) { print "$ipaddr\n"; }
}


iptables -N smtp_block
iptables -I INPUT -p tcp --dport 25 -j smtp_block
for i in `./countdenied.sh`;do iptables -I smtp_block -s ${i} -p tcp --dport 25 -j REJECT;done

Then list to see blocked hosts
iptables -nxvL smtp_block
 
You need nothing more than iptables to filter out all the connections.
 
Back
Top