SpamBlocker3.1-beta is ready

NoBaloney2

NoBaloney Internet Svcs.
Joined
Jun 17, 2007
Messages
498
Location
California
SpamBlocker3.1-beta is ready; it fixes multiple problems including the runaway spam first noted by some users approximately 10-Feb-2008.

The files and readme details may be found at:

http://www.nobaloney.net/downloads/spamblocker/DirectAdminSpamBlocker3/

Note that this release requires Dovecot and Maildir; if you haven't upgraded yet, now is the time, as it's out of beta, available in new installs, and is much more efficient with large mailboxes than is the older mbox, especially for IMAP users and anyone who stores email on the server.

Note also that the file is no longer available in different versions depending on whether or not you run ClamAV. There's only one copy of the SpamBlocker exim.conf version 3.1-beta available.

It's no longer necessary to study the ReadMe file to find all the changes you have to make to the new file; all the change information is found directly in the file itself; simply use your favorite editor to search for the word EDIT in all caps; the comment will explain all necessary and optional changes.

If you can't install it yourself, we can do it for you; please contact me at my email address below in my sig. ; Note that I don't read private messages as often as I read my email.

Jeff
 

NoBaloney2

NoBaloney Internet Svcs.
Joined
Jun 17, 2007
Messages
498
Location
California
One major change (not well doucmented) is that you can now add netblocks to /etc/virtual/bad_sender_hosts, and they will work as they should.

To resolve the issues many of us have seen over the past few days, add the following:
Code:
125.110.0.0/16
This will only work after the new SpamBlocker Version 3.1-beta exim.conf file has been installed, and it WILL solve the problem many of us are having, because the 125.110.0.0/16 netblock appears to be causing much of the current problem.

Jeff
 

chatwizrd

Verified User
Joined
Jul 3, 2005
Messages
2,007
Why does your link says 3.2 but when you go to the site its says 3.1 beta.
 

philmcdonnell

Verified User
Joined
Jan 6, 2004
Messages
182
Location
New York
One major change (not well doucmented) is that you can now add netblocks to /etc/virtual/bad_sender_hosts, and they will work as they should.

To resolve the issues many of us have seen over the past few days, add the following:
Code:
125.110.0.0/16
This will only work after the new SpamBlocker Version 3.1-beta exim.conf file has been installed, and it WILL solve the problem many of us are having, because the 125.110.0.0/16 netblock appears to be causing much of the current problem.

Jeff
What is the net block doing that it is hitting the server so hard for the last several days? Was it a botnet or something? Can we remove it later so that real emails will return?

Thanks,
Phil
 

BlueNoteWeb

Verified User
Joined
Nov 4, 2004
Messages
52
Location
Denton, TX
Jeff -

I have installed your new spamblocker file but I'm still seeing spams get through by using a fake HELO line that purports to be an IP on the server. If you'd like some log samples for analysis I would be happy to provide those to you privately.

For now I have disabled the popb4smtp system. This caused a few support tickets from users who could not send mail, but once they updated their settings to authenticate when sending everything looks to be cool.
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Great thanks Jeff! Is there a changelog as well so we can see what changed?
Unfortunately no :(. You can do a "diff" but some sections may be too far out of alignment. I was much too busy finishing it to document the changes, but it's well documented internally, including a word EDIT near every suggested/mandatory edit point.

I will document the eventual final product.

Jeff
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
What is the net block doing that it is hitting the server so hard for the last several days? Was it a botnet or something? Can we remove it later so that real emails will return?
Remove it at your own risk; if anyone is blocked by it they can contact your whitelist page :).
I'm still not sure what it was doing or why, but it was (and perhaps still is) very aggressive.

In many cases it was using an empty sender, which by RFCs we're supposed to accept since otherwise we can't get MAILER-DAEMON reports.

We were getting as much spam in a day as we usually get in a month; that's a 30x increase. I spent a lot of time making a fix.

Until we had it ready we were brute-force deleting everything in our queues every seven minutes just to give our email a minimal chance.

That's the beauty of the bad_sender_hosts file though; you can change it in realtime while watching your logfiles.

Until the next update or final release note that for highest efficiency the network-block entries should go at the top before any other entries.

Jeff
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Will the spamblocker plugin still work with this update?
When I spoke to Onno yesterday he said he wasn't sure but he is working on it now. Believe me, it's important even before SpamBlocker Plugin is ready.

Jeff
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
I have installed your new spamblocker file but I'm still seeing spams get through by using a fake HELO line that purports to be an IP on the server. If you'd like some log samples for analysis I would be happy to provide those to you privately.
I don't need any now; try this first:

The HELO line shouldn't matter if the bad_sender_hosts_ip is doing it's job; make sure you've got netblocks at the top of your /etc/virtual/bad_sender_hosts file in the format:
Code:
125.110.0.0/16
We're working on code to block on phoney HELOs but it's not quite ready yet :(. However that would replace netblocks in certain instances; the netblocks should work fine; they do for us. Don't forget to put them at the top of the file.
For now I have disabled the popb4smtp system. This caused a few support tickets from users who could not send mail, but once they updated their settings to authenticate when sending everything looks to be cool.
Again, this shouldn't matter (really) because authentication is based on the actual IP# of the server, and NOT how it HELOs.

Jeff
 

webquarry

Verified User
Joined
Mar 19, 2004
Messages
177
Hmm... after installing this version of spamblocker, I have users complaining when they run large lists on phplist (30K recipients and more) that it crawls while getting the mail out.

Indeed, the queue for exim is huge and full of their emails but the machine load is low.

Is there something in this new exim.conf that limits the number of children that can be spawned?
 

Splet

Verified User
Joined
Oct 17, 2007
Messages
17
HELO checks should permit SMTP authenticated users to send mail even if HELO is a single word. When my clients send mail via Outlook, they use their windows computer name in HELO command, which is almost always a single word.

Easy to do this in Postfix, but don't know much about Exim.
 

drewishus

Verified User
Joined
Dec 29, 2006
Messages
9
No Plans for Mbox?

Hi,
I'm looking at moving up to the 3.1 code, but saw the info regarding only support for Maildir, no mbox. Are there any plans to move support to mbox? I realize that Maildir is probably a superior way of dealing with large files, but I've got a lot of users who are using OpenWebmail, and that does not support maildir natively.
I'm hoping you'll decide to support mbox! Keep up the great work either way.
Rock,

AC
 

chatwizrd

Verified User
Joined
Jul 3, 2005
Messages
2,007
Hi,
I'm looking at moving up to the 3.1 code, but saw the info regarding only support for Maildir, no mbox. Are there any plans to move support to mbox? I realize that Maildir is probably a superior way of dealing with large files, but I've got a lot of users who are using OpenWebmail, and that does not support maildir natively.
I'm hoping you'll decide to support mbox! Keep up the great work either way.
Rock,

AC
Openwebmail can be patched to work with maildir.

Q: Does Open WebMail support maildir and vpop3mail user by Qmail?
A: The official release does still not support maildir, however,
Laurent Frigault, lfrigault.AT.users.sourceforge.net has made patched
Open WebMail 2.01 to support maildir. It is available at
http://www.agneau.org/openwebmail/openwebmail-2.01-storage-20031027.tgz
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Hmm... after installing this version of spamblocker, I have users complaining when they run large lists on phplist (30K recipients and more) that it crawls while getting the mail out.

Indeed, the queue for exim is huge and full of their emails but the machine load is low.

Is there something in this new exim.conf that limits the number of children that can be spawned?
Yes: check exim.org for the definitive information on what these limits mean:
Code:
message_size_limit = 20M
smtp_receive_timeout = 5m
smtp_accept_max = 100
message_body_visible = 3000
print_topbitchars = true
smtp_accept_max_per_host = 10
and also:
Code:
deliver_queue_load_max = 3.0
Code:
queue_only_load = 5.0
Code:
queue_run_max = 5
Note that I never optimized this code for 30,000 emails sent without limiting, and that I don't intend to. I've suggested in another thread how we handle it for our clients. DirectAdmin is meant for use in a shared environment, I use it in a shared environment, and I wrote and tested SpamBlocker exim.conf to use in a shared environment. In a shared environment availability for everyone is more important than optimization for use for any one sender.

Jeff
 

webquarry

Verified User
Joined
Mar 19, 2004
Messages
177
We have already looked at those and increased them.

The queue still takes a while to process but that is to be expected.

I'm trying to figure out why it takes so much longer for phplist to inject it's list into the queue.

The rest we can deal with.
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
HELO checks should permit SMTP authenticated users to send mail even if HELO is a single word. When my clients send mail via Outlook, they use their windows computer name in HELO command, which is almost always a single word.

Easy to do this in Postfix, but don't know much about Exim.
We currently don't check for (or against :)) single-word helo commands in SpamBlocker3.1-beta; here's the code (note that by default it's commented out):
Code:
# IF YOU CHECK FOR VALID HELO:
#  UNCOMMENT THIS SECTION
# WARNING THIS IS UNTESTED AND MAY BREAK ABILITY FOR USERS TO SEND EMAIL THROUGH YOUR SERVER
#   deny message = Single word server helo name ($sender_helo_name) rather than a FQDN.
#        condition = ${if ! match {$sender_helo_name}{\N^[^.].*\.[^.]+$\N}}
#   deny message = IP# server helo name ($sender_helo_name) rather than a FQDN.
#        condition  = ${if match {$sender_helo_name} {^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\$|^\[[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\]\$} {yes}{no}}
Note the note that uncommenting MAY BREAK ABILITY FOR USERS TO SEND EMAIL THROUGH YOUR SERVER.

Before we uncomment we'll have to accept authenticated users first; we don't do that now.

Jeff
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Are there any plans to move support to mbox? I realize that Maildir is probably a superior way of dealing with large files, but I've got a lot of users who are using OpenWebmail, and that does not support maildir natively.
I'm hoping you'll decide to support mbox! Keep up the great work either way.
Thanks for your words of support. I just don't have the time to support multiple versions of the SpamBlocker exim.conf file.

Through the release of version 2, DirectAdmin only supported mbox, so the decision was made for me.

By the time the first 3.1-beta version was out, DirectAdmin supported both mbox and Maildir, and also some folk were using ClamAV and some weren't, so I ended up supporting four versions. Big mistake. Lots of stupid mistakes every time I did an update. The long time between the first and second beta came out is directly attributable to having four versions.

Instead, beginning with 3.1-beta, I decided to go with one version of the file, with sections that users can comment/uncomment. But I didn't do a section for mbox because I no longer run an mbox-based server, and to build one to test it would have taken more time than I currently have.

If someone else does it, and can test it, and verify that it works (remember, editable sections only, not a separate file), I'll be happy to put it into the code as an EDITable section.

Of course SpamBlocker3.1-beta is my product; it's not part of DirectAdmin until/unless John and Mark decide to make it so; perhaps you should ask them; it would be quite easy for them to rewrite the section of code required and turn it over to me for inclusion.

Or you can :).

Jeff
 
Top