Exim "freaking out"

[brinkie]

I'm not sure how to explain this, but i seem to have a problem with loads of "return-mail" and spam-mails to non-existing email adresses like [email protected] (to several domains on one server) for days now. Ofcourse, there's also a lot of spam coming in to existing email adresses but that can be filtered. I even deleted one of my own sites, since it was one of the domains targeted big time (a website that has been online for almost 10 years now down for days.. ). But it doesn't seem to make much difference.

Exim is running/starting large numbers of processes (sometimes > 90, 100! slowing down the server).

Most people effected by this, have the catch all address set to ::blackhole:: so they don't have a problem with it.. But i think it is an issue that might eventually even crash the server. Server loads > 6.00 sometimes, while, in normal situations, it's 0.01-0.15

In the /input folder for exim, there is a load of mail, sometimes well over 80 Mb! I randomly checked them and in 99% it's (return)spam.

I've checked and made sure that there are no corrupt mali scripts on the server. As far as i can see, there are none. Mod_secure is installed, php's register_globals is off, etc, etc.

On my other server, with lot's more users/sites hosted, there's no such problem at all. So this lead me to believe someone was spamming using this particular server, but until now i have not found any proof for that..

Questions:
1. is there a way to prevent this?!
2. is there a way to limit the number of exim processes?
3. is there some sort of script to track what user is sending mail out? And, preferrable, how much?

I noticed that when i boot the server, the problem is gone for some time, that is the number of processes returning to normal (varying between 4 to 10). But after a few hours it starts going up again. I have 3 co-located servers, only one of them is effected big time. I've asked tech-support (hired) to look into it, but they say "it's probably a mail DDoS. It don't think so.. In that case it would continue direct after reboot, don't you think?

It is more or less driving me crazy that i can't get a grip on this.

Ps. if it helps to post (parts of) log files, server specs, etc, please let me know, i'd be happy to put it (parts) online somewhere on my server.

 

[xemaps]

just a few ideas
you can
- limit exim process to 5
- limit max conn per ip and max load
- stop using catch all !
- stop relaying
- aso ...
- suspend mail() function from php for a few days to see in log who call
- suspend forum or unsecure scripts
- add mod_load mod_evasive mod_bandwith to apache
(this is just for load which comes from httpd)

You a a lot of answer in this forum to optimize load and spam fight, and in http://www.exim.org exim manual

 

[brinkie]

just a few ideas
you can
- limit exim process to 5

Doesn't that effect delivery of mail to clients big time?!

- limit max conn per ip and max load

That makes sense. Where? In the exim.conf?

- stop using catch all !

Some people really won't be happy if i limit them in that way! In fact, some depend on that feature (catch all forwarded to their email address at their ISP). If it is send to the :blackhole: it is deleted right away, right? I've used that on the sites beeing spammed the most.. Is that what you mean?

For example, i did this on one site attracting, for some reason, a lot of spam:

2006-12-10 15:32:08 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:08 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:08 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:08 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:08 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:08 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:08 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:08 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:08 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:08 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:09 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] incomplete transaction (RSET) from <[email protected]>
2006-12-10 15:32:13 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:13 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:13 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:13 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:13 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:13 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:13 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] F=<[email protected]> rejected RCPT <[email protected]>:
2006-12-10 15:32:14 H=bzu42.internetdsl.tpnet.pl [83.19.50.42] incomplete transaction (QUIT) from <[email protected]>

But, pardon for asking, doesn't this generate bounces all the time? That's why i send it to the :blackhole: all the time (that was suggested to me some time ago).

- stop relaying

I don't get that? What do you mean with that?

- aso ...
- suspend mail() function from php for a few days to see in log who call
- suspend forum or unsecure scripts

Done that, but that didn't stop the loads of spam coming in. As i sayd, it seems it is generated because some idiots use domainnames hosted on my servers as "fake sender" for their spam..

You a a lot of answer in this forum to optimize load and spam fight, and in http://www.exim.org exim manual

Yeah, RTFM that's what i tell people too, hahaha... as if i hadn't done that already

 

[xemaps]

lot of non sense on your (bad) conf and answers
if you wan't decrease load, sorry, i can't help.
I give up.

 

[brinkie]

lot of non sense on your (bad) conf and answers
if you wan't decrease load, sorry, i can't help.
I give up.

I don't understand what's wrong with my (serious) answers/questions.

Well, anyway. Perhaps there are others that would like to try and help me. I've been using FreeBSD servers with a different (custom) control panel for 6 years and only recently switched to Linux + DirectAdmin using Exim so i guessed my questions aren't that strange. I'm not too used to mess around with mail configs, especially not Exim..

btw. i did made (some) changes on the configuration, did remove catch-alls (client level) as much as possible, did limit the number of SMTP-connections to 40, did install a script to catch spammers using malfunctioning scripts, etc, etc. --with no result-- Still, this doesn't stop tons of spam coming in (and beeing rejected and/or processed). Anyway, i'm now monitoring the servers very close since a few hours and everytime is see something suspected i change settings to see what influence this has..

All these changes did result in a nice serverload ( ~0.02-0.60) but still.. i am looking for a way to get a grip on the (exim and spamd) peaks. Only a few minutes ago, i found a spamd process going up to 96% CPU for minutes. That user in particular had spamassassin configured wrong (in conjunction with his email settings). So, step by step i'm working on it. But a faster solution would have been nice(r).

 

[nobaloney]

Since xemaps has given up I'll try to help.

I'll start at the beginning.

xemaps wrote:

- limit exim process to 5

You're right; this will drastically limit your email server's ability to work. It will make it hard for any clients using your server to send mail to connect to do so, and may result in thousands of emails waiting in your exim outgoing queue.

- limit max conn per ip and max load

Good idea on a per IP# basis; perhaps xemaps will be happy to post specific information on how to do this.

- stop using catch all !

This is quite important. I understand that some of your clients will want the convenience; they'll have to live with all the excess spam this causes them.

Note that if any of your clients are uisng AOL, Hotmail, MSN, Yahoo, and certain other large ISPs and email providers for their email accounts, and if they report all the spam to their ISP (AOL has a button making this easy), you'll be listed as the sender since your server handled the email last, and the providers will be quick to block you. You can sign up AOL so you'll get each email reported as spam from your server reported to you; this can result in hundreds of reports a day. But at least then you can search for the emails for clues as to who the culprit is and tell them to quit reporting email as spam if they're getting it from your server because they want it forwarded. One of our server clients actually cancels accounts that continue to do this after one warning.

(Note that AOL doesn't tell you who reported your server; they obfuscate that address. You'll have to trace every one of those emails through your logs to figure it out.)

- stop relaying

xemaps means stop allowing your clients to send email through your server. While this may solve some problems, it won't stop the one you're having.

- suspend mail() function from php for a few days to see in log who call

This will result in all PHP scripts on your server which send mail no longer working. If your clients use php scripts to take orders or other requests they'll be upset enough to leave. Also, it won't stop the problem you're having.

- suspend forum or unsecure scripts

In spite of those hundreds of emails I highly recommend signing up with AOL to get them; visit: http://postmaster.aol.com/ and click on Would you like to setup a feedback loop?.

Because if you've got any insecure forms on your server they'll sooner (rather than later) start sending spam to AOL members, and this is your early warning that your clients have insecure scripts.

Yes, this can result in you spending a few hours a day scanning emails from AOL to see if any of them have lists of many recipient addresses. I'm sorry if anyone told you that webhosting was easy and didn't require much time .

- add mod_load mod_evasive mod_bandwith to apache
(this is just for load which comes from httpd)

Neither of these will affect spam.

Now, as to what you wrote:

I'm not sure how to explain this, but i seem to have a problem with loads of "return-mail" and spam-mails to non-existing email adresses like [email protected] (to several domains on one server) for days now.

The explanation is really quite simple: the return emails are coming from poorly configured servers who instead of refusing email destined for non-existent users first accept it and then send it back. Those servers are misconfigured and you could block them all, but if you do you may end up blocking some servers you should probably want to reach.

Here's what's happening:

Spammers pick return addresses at random:

Some poor soul has an infected Windows system somewhere (I read recently that spammers add roughly 150,000 infected Windows systems daily to their arsenal). That machine has an addressbook on it. The spam software installed on that infected system sends spam from a random address.

That address is at a domain belonging to one of your users.

The spam is sent to lots of addresses, including lots of nonexistent addresses.

Properly configured servers will refuse the email because the recipient address doesn't exist.

But improperly configured addresses (see above) will accept it, find out the address doesn't exist, and send it back to the from address or the envelope sender address.

Which has been forged to be a domain for one of your users.

And your server gets the mail back as returned email.

Getting rid of those catchall addresses should help.

Ofcourse, there's also a lot of spam coming in to existing email adresses but that can be filtered. I even deleted one of my own sites, since it was one of the domains targeted big time (a website that has been online for almost 10 years now down for days.. ). But it doesn't seem to make much difference.

If the users don't exist then does the email still come in? It shouldn't.

Exim is running/starting large numbers of processes (sometimes > 90, 100! slowing down the server).

You can decide to allow that or drop the number of allowed processes (see way above).

Most people effected by this, have the catch all address set to ::blackhole:: so they don't have a problem with it.. But i think it is an issue that might eventually even crash the server.

Which is why I prefer refusing the mail to blackholing it. That way your server won't spend any more time on it once it identifies the address is incorrect.

Server loads > 6.00 sometimes, while, in normal situations, it's 0.01-0.15

If your server's got enough memory and is fast enough 6.00 shouldn't kill it but it will slow things down.

In the /input folder for exim, there is a load of mail, sometimes well over 80 Mb! I randomly checked them and in 99% it's (return)spam.

What /input folder?

I've checked and made sure that there are no corrupt mali scripts on the server. As far as i can see, there are none. Mod_secure is installed, php's register_globals is off, etc, etc.

I explained above how to use AOL to help you find bad mail scripts.

On my other server, with lot's more users/sites hosted, there's no such problem at all. So this lead me to believe someone was spamming using this particular server, but until now i have not found any proof for that..

It's possible, but unlikely. If someone was you'd see the outgoing mail in your logs.

Unless of course someone has uploaded their own mail server .

3. is there some sort of script to track what user is sending mail out? And, preferrable, how much?

There is a patch. I can't find it right now ;( .

I noticed that when i boot the server, the problem is gone for some time, that is the number of processes returning to normal (varying between 4 to 10). But after a few hours it starts going up again.

My guess is that those servers have stopped trying while your server was down, and they've set their own internal try-again clock.

I have 3 co-located servers, only one of them is effected big time.

Consider yourself lucky.

I've asked tech-support (hired) to look into it, but they say "it's probably a mail DDoS. It don't think so.. In that case it would continue direct after reboot, don't you think?

It's certainly effectively an email DDos. But I think it's happening for reasons I've mentioned above. Mail servers will stop trying for an arbitrary length of time after they get a failure to connect, so that's why I think that's what it is.

It is more or less driving me crazy that i can't get a grip on this.

My guess is you'll do a lot better when you start refusing catchall (fail) rather than blackhole.

Ps. if it helps to post (parts of) log files, server specs, etc, please let me know, i'd be happy to put it (parts) online somewhere on my server.

Please no, unless someone asks for it. I tend to go blind trying to read large log excerpts.

Jeff

 

[brinkie]

Jeff,

thanks for taking so much time and explaining in such detail, this really helped! I will look into the aol link a.s.a.p.

It seems that the following worked for me. For a start..

I've limited the number of connections per ip/host in the exim.conf and made some other changes.

smtp_accept_max = 40

I know this is 'rude' but there are about 200 users on that server. I don't think there are more than 40 of them trying to send at the same time.. will see what happens using this setting. Until now (a few days) i've received no complaints.

I also added these lines, and it did a wonderfull job..

smtp_accept_max_per_host = 10
smtp_accept_queue_per_connection = 20

We'll see who will start complaining about this.. If they do, changes are they are using the mail for things i don't like at all I'll be playing around with these settings until i find the perfect mix.

Catch-all has been switched off on all client sites, accept for two or three.

 

[polo]

Just starting having the same thing happen, I've disabled all catch-alls and waiting for the slow down to begin.

Jeff did you ever find that script

3. is there some sort of script to track what user is sending mail out? And, preferrable, how much

?

There is a patch. I can't find it right now ;( .