SpamBlocker 4.3.0, BlockCracking, Easy Spam Figther, and new exim.pl

Thanks for the report.
I've added code for ESF 1.4.

New changes:

1) variables.conf has:
Code: 
[COLOR=#000000]EASY_SKIP_SENDERS = /etc/virtual/esf_skip_senders
[/COLOR][COLOR=#000000]EASY_SKIP_RECIPIENTS = /etc/virtual/esf_skip_recipients
[/COLOR][COLOR=#000000]addresslist esf_skip_senders = ${if exists{EASY_SKIP_SENDERS}{wildlsearch;EASY_SKIP_SENDERS}}
[/COLOR][COLOR=#000000]addresslist esf_skip_recipients = ${if exists{EASY_SKIP_RECIPIENTS}{wildlsearch;EASY_SKIP_RECIPIENTS}}
[/COLOR]which is used by the other configs to skip things, accordingly.

2) I've changed the check_dkim.conf file to "warn" instead of deny.
The variables.conf has a new setting:
Code: 
[/FONT][FONT=Verdana]EASY_DKIM_FAIL = 100
and will add the score to the total:
Code: 
[/FONT][FONT=Verdana]  warn
[/FONT][FONT=Verdana]    dkim_status = fail
[/FONT][FONT=Verdana]    set acl_m_easy69 = ${eval:$acl_m_easy69+EASY_DKIM_FAIL}
[/FONT][FONT=Verdana]    add_header = DKIMCheck: Server failed DKIM test, EASY_DKIM_FAIL Spam score
[/FONT][FONT=Verdana]    log_message = DKIM: Failed. reason='$dkim_verify_reason'
which lets you set a lower score, rather than an immediate drop for DKIM Failures.

John
 
zEitEr said:
With exim.conf 4.3.2 you can use whitelists to skip some tests, including DKIM check.
So all what is needed is to add trusted IPs into /etc/virtual/whitelist_hosts_ip (one IP per line with empty bottom line).

At least this is what we started to use. I hope we could disable test other way.
Click to expand...
Just be careful with whitelist_host_*... as it also means the IP can relay email through your server.

The new esf_skip_senders and esf_skip_recipients shoudn't bypass any of the usual things like auth. They'll only skip ESF checks: SPF, DKIM, and ESF's RBL (assuming I did it correctly).

John
 
Hello John,

Thank you for your tip. I've changed server to use /etc/virtual/esf_skip_recipients instead of /etc/virtual/whitelist_hosts_ip
 
1.4 was a little buggy. 1.5 was a quick fix (recipients ACL condition is only valid in the RCPT ACL, so I quickly removed it)

I've made more changes and released 1.6.
I've added /etc/virtual/esf_skip_hosts, so yo can fill up server.domain.com values there, or even *.domain.com (wildlsearch works).
Saves needing to use whitelist_hosts.
I didn't add esf_skip_hosts_ip. If the IP connecting to you doesn't have a host, just add something to /etc/hosts, so that it does.

1.6 now uses a variable (acl_m_esf_skip), set to 1 wherever the esf_skip_* file is loaded in and compared, to tell the final check in check_message.conf to skip the "drop" sections.
All other parts are still run, given scores.. but a high score doesn't matter now... as the message won't be dropped.

There are some exceptions:
- The esf_skip_recipients will not bypass a failed SPF record because the SPF check is done before the RCPT TO (recipient), and is after the MAIL FROM. Exim doesn't know the recipients yet.
- so for SPF checks, only these two work: use esf_skip_hosts or esf_skip_senders.

John
 
Edit: This might be related to RBLs, not ESF. I added the domain to /etc/virtual/whitelist_domains and it's quick again.

Since implementing this, SMTP accepts mail very slowly. I have a bulk mailer that opens an SMTP connection (to localhost) and then sends ~100 messages. This was taking about 10 seconds to run (slightly longer to deliver). Now the SMTP connection is open for about 15 minutes; Exim is accepting a message approximately every 10 seconds.

I assume the slowdown is in the ESF checks, so I added the domain associated with this mailer to /etc/virtual/esf_skip_senders with a wildcard:
Code: 
*@example.com

This didn't help. Mail is still accepted very slowly.

Am I whitelisting correctly?

Is there a way to profile Exim and figure out exactly what is taking so long? (I assume it's waiting for DNS queries to come back)
 
Last edited:
Wouldn't this be a good time to ditch the:

Code: 
#EDIT#5:
system_filter = /etc/system_filter.exim

part? Since there is already so much filtering done elsewhere and that filter set looks to be almost 13 years old now.
 
Last edited:
Open to discussions, but I personally still think people shouldn't be sending exe/.doc/dangerous files around :)
This is where those would be blocked.
If you have a specific file or rule you think should be a allowed, let us know.

John
 
I don't recall now where it is but I know that we had to remove the block of attached .eml files because that's how RoundCube forwards multiple emails.

And... the one thing keeping me from using your new SpamBlocker file was that (the last time I looked) it didn't support Dovecot delivery. Does it now? I need Dovecot delivery so I can I can use Sieve. If it uses Dovecot delivery (at least as an automated option through CustomBuild 2) then I can finally start shutting down my own version of SpamBlocker in favor of the new one.

Jeff
 
DirectAdmin Support said:
Open to discussions, but I personally still think people shouldn't be sending exe/.doc/dangerous files around :)
This is where those would be blocked.
If you have a specific file or rule you think should be a allowed, let us know.

John
Click to expand...

John / tristan,

Here we don't use system_filter.exim and we block dangerous attachments directly at exim.conf:

Code: 
  ## deny  if email contains an attachment of type we don't accept.
  deny message = 554 denied. 5.7.1 Rejected for containing prohibited attachment (.$found_extension)
  demime = bat:cmd:com:cpl:crt:exe:hta:inf:ins:isp:js:jse:lnk:msc:msi:msp:pif:prf:reg:scr:sct:shs:vb:vbs:vbe:wsh:wsc
 
nobaloney said:
And... the one thing keeping me from using your new SpamBlocker file was that (the last time I looked) it didn't support Dovecot delivery. Does it now? I need Dovecot delivery so I can I can use Sieve. If it uses Dovecot delivery (at least as an automated option through CustomBuild 2) then I can finally start shutting down my own version of SpamBlocker in favor of the new one.
Click to expand...
Yes, since 4.3.0.

John
 
ditto said:
Regarding forwarders in DirectAdmin. When customer forwards emails to for example their gmail accounts, then spam is also forwarded. Gmail see the spam as if it is sent from our mail server.

I ask that DirectAdmin can change the forwarders so that the envelope sender is not changed when forwarding emails, so that Gmail don't see it as spam sent from our mail server. Here is a quote from "Best practices for forwarding mail to Gmail" at https://support.google.com/mail/answer/175365?hl=en
Click to expand...

twv said:
Not sure if this is the right thread for this request, but I'd like to add my agreement. A couple of clients get LOTS of spam and they insist on forwarding everything to their gmail accounts, so gmail frequently gives warnings about rate-limiting our server.
Click to expand...

I agree. Sometimes messages are rejected at the server where the forward is located to. Eg. someone sends an email to a domain on our server and the message gets forwarded, the destination server can reject this message for several reasons. Local blacklist of our server, rate limiting, but also SPF. Sometimes the senders domain has an SPF record with fail option (-) configured. The IP address of our server isn't configured in the SPF record and therefore the destination server rejects the email forwarded through our server.

I think it's best to keep the envelope sender to prevent all these problems.
 
unihostbrasil said:
John / tristan,

Here we don't use system_filter.exim and we block dangerous attachments directly at exim.conf:

Code: 
  ## deny  if email contains an attachment of type we don't accept.
  deny message = 554 denied. 5.7.1 Rejected for containing prohibited attachment (.$found_extension)
  demime = bat:cmd:com:cpl:crt:exe:hta:inf:ins:isp:js:jse:lnk:msc:msi:msp:pif:prf:reg:scr:sct:shs:vb:vbs:vbe:wsh:wsc
Click to expand...

Hi John,

I thinks Alex' code would work a lot more transparently then to link to this very very old version of the system_filter. Because of the regex in that file there are quite some possibilities for false positives, with files containing spaces, dots and combination of dangerous file extension in the name of the attachment for example.
 
Besides the attachment block, there is still one piece that I feel is relevant in system_filter.exim (even if it's something that was added into that file afterwards, rather than included by default).

Code: 
if $sender_address is ""
then
        if $h_auto-submitted is not "auto-replied"
        then
           if ${lookup{${extract{2}{@}{$recipients}}}lsearch{/etc/virtual/domains}{yes}{no}} is "no"
           then
                noerror fail text "Delayed bounce message ignored"
                seen finish
           endif
        endif
endif

That cuts down on the number of e-mails we handle which have an empty return path. The only problem we've ever experienced with it is that without the second if statement in place, it will prevent auto-responders and similar from functioning. Checking for auto-replied allows for auto-responders that were set up via DA to still go through but we also use the vacation portion of managesieve to allow users to create autoresponders via Roundcube. Those use a slightly different wording for their auto-submitted header which also needs to be checked for. It's something that would likely take some brief investigate work to resolve but I haven't been able to touch it yet.

If there is an alternative and cleaner way of handling that though, I'd be all for scrapping the filter.
 
Any idea why the setting split_spool_directory=yes was removed since SB 4.x ?

I think it is a very handy feature: it speeds up mail delivery (according to DirectAdmin) and in case a server is on a spam run, less experienced users can have trouble with cleaning the mail queue as rm will not take more than xxxxx files.
 
I agree, and I use it on my own machines. John, please consider this a feature request.

Though it's not in my published masters (John, is this the reason you don't include it?) it IS in the installations on my own servers.

You may need to keep this as an option (default no) to maintain backwards compatibility, but I do like it.

Jeff
 
I'm totally "for" the split spool.. I'll add it in... probably got missed when we adopted 4.x.
Thanks for the report.

John
 
Will adding the split-pool cause exim to automatically install the directories, or will this require manual work? And will it be required that any files in current spool system be moved to proper subdirectories?

Jeff
 
From the exim docs:
Code: 
[COLOR=#000000][FONT=Verdana]It is not necessary to take any special action for existing messages when changing [/FONT][/COLOR][COLOR=#000000][FONT=Verdana][B]split_spool_directory[/B][/FONT][/COLOR][COLOR=#000000][FONT=Verdana]. Exim notices messages that are in the “wrong” place, and continues to process them.[/FONT][/COLOR]
John
 
Goodmorning!

So i'm running this for a time on a server of mine for testing.

It worked perfectly for the past few weeks.
Spam has been reduced alot.

But the last few days i'm getting a high server load caused by Spamd
It' holds on for around 10 a 15 minutes and then it goes back to normal.
And after 1 a 2 hours it's back again.

So i don't know if it's caused by this new "function" or something else.

Output from PS:
Code: 
 USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

 root     23947  0.0  0.6 244892 55240 ?        Ss   07:14   0:05 /usr/bin/spamd -d -c -m 15
 nobody   23950  6.4  1.7 331816 141888 ?       R    07:14   7:27  \_ spamd child
 nobody   23951  3.5  1.6 322812 133044 ?       R    07:14   4:08  \_ spamd child
 nobody   32595 36.9  1.6 319556 129600 ?       R    09:06   1:24  \_ spamd child
 nobody   32596 42.7  1.5 319296 128344 ?       R    09:06   1:37  \_ spamd child
 nobody   32597 27.1  1.6 326196 136508 ?       R    09:06   1:02  \_ spamd child
 nobody   32598 40.6  1.5 319300 128352 ?       R    09:06   1:33  \_ spamd child
 nobody   32599 50.8  1.5 319300 128352 ?       R    09:06   1:56  \_ spamd child
 nobody   32600 44.1  1.5 319300 128336 ?       R    09:06   1:41  \_ spamd child

Configuration:
Exim: 4.85
Exim.conf: 4.3.2
Exim.pl: 19-alpha2
BlockCracking: 1.1
EasySpamFighter: 1.6
Spamassassin: 3.4.0
ClamAV 0.98.6

Steps tried:
cd /usr/local/directadmin/custombuild
./build set eximconf yes
./build set eximconf_release 4.3
./build set blockcracking yes
./build set easy_spam_fighter yes
./build set spamassassin yes
./build update
./build exim_conf
./build exim
./build spamassassin

But the problem is still existing on the server.
 
Spamassassin consumes a lot of CPU, especially when analyzing large emails. Maybe you are receiving many emails with large attachments.

Check if your "spamcheck_director" at exim.conf or exim.spamassassin.conf have this rule, and try lowering it (here we just analyse messages under 200k):

Code: 
{<{$message_size}{500k}} \

Also make sure that the high CPU consumption is not a I/O or memory bottleneck on your server.

Be sure that you are using the updated rules of Spamassassin:

/usr/bin/sa-update -D && killall spamd && /usr/bin/spamd -d -c -m 15
Click to expand...
 
You must log in or register to reply here.
Back
Top