Need a permanent solution: Email BSMTP 421 Errors

hostpc.com

hostpc.com

Verified User
Joined
Aug 2, 2003
Messages
1,054
Location
Schenectady, NY
Error message:
An error was detected while processing a file of BSMTP input.
The error message was:

421 SMTP incoming data timeout - message abandoned

The SMTP transaction started in line 0.
The error was detected in line 3.
0 previous messages were successfully processed.
The rest of the batch was abandoned.
421 SMTP incoming data timeout - message abandoned
Transaction started in line 0
Error detected in line 3


Using latest exim.conf
Using SpamAssassin: SpamAssassin 3.0.2

We're (our customers) seeing these errors more and more.

I've done some homework:

I believe this error is only generated when there is more than one recipient specified in the email.

This error is caused (it's commonly believed) by a combination of Exim and SpamAssassin. We're running the most current stable versions of both packages. It appears that this issue has existed in the last (at least) 3 releases of Exim - and remains unfixed. On the cpanel forums there are many supposed "patches" and "fixes" - most of them have failed within hours of trying. I haven't fouond one valid combination that appears to FIX permanently the error.

With that said, we have 3 options. The first would be to eliminate SpamAssassin and hope that fixes the issue (frankly I'm not 100% convinced it will). The second option is to ignore the errors, had ask you to send mail a little later, or send it through your ISP, or 3rd party account such as gmail, etc. The third option is to hold out, ignore these errors (send through your ISP or retry on our servers later) - and hope a permanent solution is released soon.

Due to the amount of spam out there, I'm hesitant to disable SpamAssassin.

There's a ton of information on Google concerning this - all points to errors in Exim.

There's been several threads here concerning this error - and thousands more on sites across the net.

If Exim is indeed the problem - maybe it's time to consider another mail daemon?

Any info you can provide would be appreciated.

Thanks

Joe
 
Joe, I found this right here on DA forums:

Thanks for your reply! It seems to be a BSMTP and Spam Assassin error when spamc is not running

So it's got nothing to do with Exim, but rather occurs when exim can't contact Spam Assassin.

Jeff
 
Ok, lets say for the sake of argument that SA is the culprit.

What's the solution for that? Disable SA? I can't even imagine the flood of spam if I did that.

You're the exim God around here - there has to be a solution
 
My understanding is that spamd runs all the time and calls spamc.

Am I right? Anyone?

Anyway, on my dedicated rental servers we offer enough horsepower and generally SpamAssassin continues to run, and everything works.

On my personal server I did end up turning off SA (so I can see how good SpamBlocker works), and I found that SA was only catching about ten emails a day why SpamBlocker catches hundreds to thousands.

To put it another way, of the emails that came through SpamBlocker, SpamAssassin was only catching about a third.

Jeff
 
jlasman said:
Anyway, on my dedicated rental servers we offer enough horsepower and generally SpamAssassin continues to run, and everything works.
Jeff
Click to expand...

Our servers are all well optimized Dell servers, p4 2.8 or higher with 1GB of RAM ... all have CPU loads of 0.1 or less, 24x7 (exept during backups) - so I dont think "horsepower" has anything to do with it - do you agree?
 
Jeff, no offense meant to you at all, but I can't, in good conscious tell my users that they need to go from using spamassassin (which lets every email get evaluated and delivered if the SA rating is high enough (user controlled)) to using this system that requires senders clicking on a web link to verify their identity...

If some spammer did see the error message s/he wouldn't bother to follow an html link to learn how to unblock his/her address; it's just not worth it to the her/him for a one time email delivery when s/he's got tens of millions of other addresses.
Click to expand...

Several people, especially on dynamic IP's (dialup) may get false positives if the IP leasee previous to them got that IP listed at any RBL.

Some customers might go for that method, but when you take the ability out of the users hands, they're gonna get PO'd.

There has to be another way around this.

DA provides Exim, and includes SpamAssassin in their release (which ironically was written by you- enabled) - I think DA needs to step up and find a solution on this matter.

Joe
 
I have to agree with Joe.

The fact is, this effects to bottom line in that email is without the doubt the MOST important aspect of hosting IMHO. Clients will be forgiving on a lot of things like short amounts of downtime, but if their email does not work...it's an enormous issue. One that loses customers.

I think either this problem needs to be solved pronto.
 
No, I had nothing to do with SpamAssassin. I wrote SpamBlocker.

I would love SpamAssassin if it worked. I'd even use it.

But the fact remains that as spammers continue to inundate us with more and more spam, using SpamAssassin becomes a game of catch-up, and we continue to need servers with more and more horsepower to keep up.

It's not really a DA issue if you're getting so much spam that spamd won't keep running. If you think it is, I'd love to know your reasoning.

Joe, I understand your issues, your concerns, and your angst.

I hang around various forums where the big guys hang out... the ones who handle tens of thousands of mailboxes. They complain all the time about how much horsepower they have to add to their email servers because of increases in the amount of spam coming in.

The issue is quite simple... spam now (for us anyway) makes up about 90% of the email traffic.

Spam Assassin requires that every email be read and managed. By a perl script. That's just not efficient.

Is there a more efficient way? Yes; it's called reputation. Which is what SpamBlocker uses. A recent issue of InfoWorld (a U.S.-based magazine covering IT and internet issues) ran an issue dedicated to covering Anti-Spam solutions. Every one of them used reputation. And InfoWorld liked that method for reasons you may be able to find looking up the reviews on their website.

Joe, I disagree with you about how most people feel about spam. In my opinion most people would rather stop looking through those spams to see if they're really spam or not, because if they do look through them they're spending just as much time as if they didn't have SpamAssassin. However they can't. They must look through them because otherwise they'll lose important email.

With blocking solutions we find we don't lose important email.

The reason is simple. Because we refuse email from sending servers with a bad reputation. Because we refuse the email the sender gets it back, with instructions as to how to get whitelisted.

However if you disagree with me and either don't like the solution, or you really believe your users won't like it, then go ahead and:

1) complain to the authors of SpamAssassin; tell them that their software just can't handle all the spam you're getting without crashing.

2) complain to the author of the exim.conf file, trying to convince him that the problem only occurs because of the way the file calls SpamAssassin.

or

3) throw more hardware at the problem.

I truly believe you can change mailservers and control panels all day without resolving the problem.

Perhaps this discussion better belongs on the SpamAssassin mailing list, but I'm not going to take it there because my clients are very happy we block the bad stuff and if one of their correspondents happens to be accidentally listed as bad, we send the email back so the sender can ask to be whitelisted.

I know it's not a perfect solution. Frankly, I don't think there is one.

If you do, then please either write it or have it written, and I'll be happy to implement it on my servers :) .

Jeff
 
sullise said:
I think either this problem needs to be solved pronto.
Click to expand...
Sean, please give me some idea as to how you'd suggest any one of us solve it?

Parts of SpamAssassin are already compiled. Should we rewrite SpamAssassin completely in C++ or some other more efficient language to make it more responsive? I can't do that; perhaps someone else has the time, the energy, and the interest.

Should we throw more hardware at our servers? I can only do that on my machines. (Well I could throw hardware at your machine, but it might break by the time you got it, and if it didn't you'd still have to install it :D .)

Should we cut down on the amount of spam being sent? Since I'm not responsible for sending spam, I can't do that. I don't think any of us are sending the spam, so I don't think any of us can do that.

Or should we cut down on the amount of spam we accept? (This is what SpamBlocker does.)

Personally I've already mustered up all the memory of college logic classes I can, and that last option seems to me to be the only one I have any reasonable control over. So that's the direction I've taken.

Please, please, please, come up with something else if you can.

Thanks.

Jeff
 
my spamd processes dont fail - I'm not sure where that assumption came from. In fact, what fails for one user, 10 seconds later may work 100% fine.

For instance, there's one shared server with 8 domains on it. Fact is, it happens more often on there than any other server (with up to 200 on it). Again - it will throw the BSMTP error now, and 2 minutes later be 100% fine.
 
jlasman said:
Sean, please give me some idea as to how you'd suggest any one of us solve it?
Click to expand...

I'm not guru enough to offer any suggestions other then dumping mbox and going with maildir.

This thread I found HERE pretty much gives a good alternative that DA seems to be ignoring. While it may not be perfect, and MAY not solve THIS particular problem definitely would cut down on a lot of other IMAP issues.

http://www.directadmin.com/forum/sh...page=20&highlight=replacing exim&pagenumber=3
 
I found an interesting mailing list post here; I'm going to ask DA to take a look at it.

hostpc.com:

Perhaps I'm wrong, but reading about the relationship between spamc and spamd here I get the understanding that spamc fails because spamd isn't running. In any event, it's a SpamAssassin problem. Unless it's got something to do with the mailing list post I made above I don't see how it could have anything to do with either exim or with DA.

That said, if I were using SpamAssassin myself I'd probably be posting on the SpamAssassin mailing list and asking the questions there.

Sean:

I'm a firm believer in switching to Maildir, but it won't do a thing about problems with SpamAssassin.

That said, if you read the thread you'll notice that I offer a testbed server on a dedicated IP#, with an installed DA license, to anyone willing to work on it for the good of the community.

That offer still stands

Jeff
 
I've never seen this myself, but it does appear to be a pretty common problem. First, I would upgrade to the latest SA and make sure that your perl modules are current as well. Then, there's a couple of things I would try:

1) Increase the number of spamd children that are spawned. The -m 5 switch in the exim initscript should do this.
2) Use unix sockets for SA. Add a --socketpath to spamd in the exim initscript and add a -U switch to spamc in exim.conf.
3) Try to recreate the failure manually. That is, schedule a job that will cat an email through sa via spamc and watch the output and time. At some point, you should find that it returns 2-3 lines and either pauses more than 5 minutes or exits or shows an error or something.
4) Run spamd and spamc in debug and watch for errors that coincide with the exim error.

There are also a couple of other things you can do:

1) Allow the message to defer if the filter returns a failure by removing return_fail_output and adding a temp_errors switch instead. You may want to do some logging in the transport as well.

2) Do SA checking in the ACL instead of in the director/transport. I've looked at this for a bit, but it appears that some flexibility is lost using this method.
 
ballyn said:
1) Increase the number of spamd children that are spawned. The -m 5 switch in the exim initscript should do this.
2) Use unix sockets for SA. Add a --socketpath to spamd in the exim initscript and add a -U switch to spamc in exim.conf.
Click to expand...
ballyn, do you recommend that these become standard for DA?

Jeff
 
I haven't had a problem with 5 children, but busy sites might. I would probably suggest sockets as the default since we're not using multiple spamd targets.
 
Forgive me, I"m but a humble servent ... by " exim initscript" - do you mean exim.conf ?

I'm confused where to make these changes you speak of :)

Joe
 
In /etc/init.d/exim:
Code: 
if [ -e /usr/bin/spamd ]; then /usr/bin/spamd -d -c -m 10 --socketpath=/var/spool/spamd/spamd 1>/dev/null 2>/dev/null; fi
In /etc/exim.conf:
Code: 
transport_filter = /usr/bin/spamc -U /var/spool/spamd/spamd -u ${lookup{$domain}lsearch*{/etc/virtual/domainowners}{$value}}
 
Joe,

Please keep us posted as to whether or not this helps; I'd love to start using SpamAssassin again on my personal domains.

Thanks.

Jeff
 
I just tried it on two of the most problematic (more than a couple reports) - lemme give it a day or two and I'll report back here.
 
You must log in or register to reply here.
Back
Top