SpamBlocker 4.3.0, BlockCracking, Easy Spam Figther, and new exim.pl

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,793
Location
A Coruña, Spain
Yes, the problem was that the script i'd made was making a single place where to save bayes_* files in /root/.spamassassin since the sa-learn command was running at root level.

Since there are many accounts and many emails, the files was growing very fast and the sa-learn command was going to take all disk I/O causing server incredible slow.

I've fixed the script now, at this moment the script run as root but run the sa-learn command at user level, so the /home/USER/.spamassassin folder is used for bayes and for user_pref file.

At this time, the script check for Maildir/.Junk/cur-new for Spam and Maildir/cur-new for HAM (taken from INBOX itself, that's why files grow, if user doesnt use other folder than INBOX the scan would take more)

DA for now use Maildir/.INBOX.Spam as default folder for Spam, but i know that John is studing a way to change that in DA code (i suppose) to make it editbale, or at least to change it to Mildir/.Junk (since is the default folder used from most client for that purpose).

The script is this one:

Code:
#!/bin/sh

###############################################################################
#                                                                             #
#                                Crazy Network                                #
#                                                                             #
#                           http://www.CrazyNetwork.it                        #
#                                                                             #
#                              Info@CrazyNetwork.it                           #
#                                                                             #
###############################################################################

if [ "$1" != "" ]; then

        if [ -d /usr/local/directadmin/data/users/$1/ ]; then

                COMMAND="echo $1"

        else

                echo "Invalid user"

        fi

else

	COMMAND="ls /usr/local/directadmin/data/users/"

fi

for DA_USER in `$COMMAND`

	do
	echo ""
	echo "#########################################################################################"
	echo "Starting user $DA_USER"
	echo ""
	DA_HOME=/home/${DA_USER}

	touch $DA_HOME/sa-learn.log
	chown $DA_USER:$DA_USER $DA_HOME/sa-learn.log

	#set this to 1 if you want the spam be removed after the run
	DELETE_SPAM_DATA=0

	learn_Maildir()

		{
		FILESPAM=${1}/.Junk
		FILEHAM=${1}
		ACCOUNT="`echo $maildir | cut -d/ -f6`@$d"

		if [ "$ACCOUNT" = "@" ]; then

			ACCOUNT="$DA_USER@$HOSTNAME"

		fi

		if [ -e ${FILESPAM}/new ] || [ -e ${FILESPAM}/cur ]; then

			echo "Learning SPAM for account ${ACCOUNT}";
			su - $DA_USER -s /bin/sh -c "nice sa-learn --no-sync --spam ${FILESPAM}/{cur,new} > ${DA_HOME}/sa-learn.log" >/dev/null
			cat $DA_HOME/sa-learn.log

		fi

		if [ -e ${FILEHAM}/new ] || [ -e ${FILEHAM}/cur ]; then

			echo "Learning HAM for account $ACCOUNT";
			su - $DA_USER -s /bin/sh -c "nice sa-learn --no-sync --ham ${FILEHAM}/{cur,new} > ${DA_HOME}/sa-learn.log" >/dev/null
			cat $DA_HOME/sa-learn.log

		fi

		if [ "$DELETE_SPAM_DATA" -eq 1 ]; then

			rm -f ${FILESPAM}/new/*

		fi

	}

	if [ -e $DA_HOME/Maildir ]; then

		echo "Starting main E-Mail account for user $DA_USER"
		echo "-----------------------------------------------------------------------------------------"
		learn_Maildir $DA_HOME/Maildir
		echo "-----------------------------------------------------------------------------------------"
		echo ""

	fi

	for d in `ls $DA_HOME/imap`; do

		{

		> $DA_HOME/sa-learn.log
		echo "Starting E-Mail accounts for domain $d"
		echo "-----------------------------------------------------------------------------------------"

		DOMAIN_DIR=${DA_HOME}/imap/${d}
		if [ -h $DOMAIN_DIR ]; then

			echo "No E-Mail accounts for domain $d"
			echo "-----------------------------------------------------------------------------------------"
			echo ""
			continue;

		fi

		for maildir in `ls -d ${DOMAIN_DIR}/*/Maildir 2>/dev/null`; do
		{

			learn_Maildir ${maildir}

		};

		done;

		if [ "`cat $DA_HOME/sa-learn.log`" = "" ]; then

			echo "No E-Mail accounts for domain $d"

		fi

		su - $DA_USER -s /bin/sh -c "nice sa-learn --sync" >/dev/null
		echo "-----------------------------------------------------------------------------------------"
		echo ""

		};

	done;

	echo "Statistics for user $DA_USER:"
	echo "-----------------------------------------------------------------------------------------"
	su - $DA_USER -s /bin/sh -c "nice sa-learn --dump magic > $DA_HOME/sa-learn.log" >/dev/null
	cat $DA_HOME/sa-learn.log
	echo "-----------------------------------------------------------------------------------------"
	rm -rf $DA_HOME/sa-learn.log
	echo "#########################################################################################"

done;

exit 0;
Hope you enjoy it :) Feel free to edit it as you need to, specially for the Spam folder changing:
FILESPAM=${1}/.Junk
to
FILESPAM=${1}/.INBOX.spam
Also, if you want a folder different from INBOX as folder for teach the non-Spam change:
FILEHAM=${1}
to
FILEHAM=${1}/.NON-SPAM-FOLDER

USAGE:
- Save it as filename you prefer (ex. script.sh)
- Run it with no extra info will run on all users (ex ./script.sh)
- Run it with username as extrafilm will run the script just for specified user (ex. ./script.sh admin)

Regards
 
Last edited:

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Thanks for posting the script, Andrea. Please post again in a few days with notes that everything works, and then I will probably give it a try :).

I don't think I'd want it to work on a folder named Junk I know that I often put stuff that isn't spam, but that I just don't want, or stuff I'm not sure about, in my Junk folder, but only spam I'm certain is Spam in my Spam folder.

Jeff
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,793
Location
A Coruña, Spain
Also, if you want a folder different from INBOX as folder for teach the non-Spam change:
FILEHAM=${1}
to
FILEHAM=${1}/.NON-SPAM-FOLDER

Note: added this post to the script post to have all info's on a single post.

Regards
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,793
Location
A Coruña, Spain
Sample output:

Code:
>./spamassassin_teach.sh testserver

#########################################################################################
Starting user testserver

-----------------------------------------------------------------------------------------
Learning SPAM for account testserver@server.test-server.it
Learned tokens from 0 message(s) (0 message(s) examined)
Learning HAM for account testserver@server.test-server.it
Learned tokens from 0 message(s) (0 message(s) examined)
-----------------------------------------------------------------------------------------

Starting E-Mail accounts for domain test-server.it
-----------------------------------------------------------------------------------------
Learning SPAM for account andrea@test-server.it
Learned tokens from 0 message(s) (0 message(s) examined)
Learning HAM for account andrea@test-server.it
Learned tokens from 0 message(s) (6 message(s) examined)
-----------------------------------------------------------------------------------------

Statistics for user testserver:
-----------------------------------------------------------------------------------------
0.000          0          3          0  non-token data: bayes db version
0.000          0          0          0  non-token data: nspam
0.000          0          6          0  non-token data: nham
0.000          0        536          0  non-token data: ntokens
0.000          0 1406795158          0  non-token data: oldest atime
0.000          0 1412853644          0  non-token data: newest atime
0.000          0 1414679145          0  non-token data: last journal sync atime
0.000          0          0          0  non-token data: last expiry atime
0.000          0          0          0  non-token data: last expire atime delta
0.000          0          0          0  non-token data: last expire reduction count
-----------------------------------------------------------------------------------------
#########################################################################################
 
Last edited:

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Thanks, Andrea. I think I can likely use this with minor change:

My understanding is that if something is not spam you don't need to teach SpamAssassin anything about it; only for email incorrectly classified as SPAM.

I'm thinking have SpamAssassin do it's automatic classification and put all spam in junk (set as the spam special folder). Then I can manually create two folders: spam and ham.

Then if I find something gets through that should be classified as spam I can put it in the spam folder, and if I find something gets into the junk folder that shouldn't have been classified as spam I can temporarily put it into the ham folder. Then I can run the script manually nightly to teach SpamAssassin, and every so often delete the spam and move the ham email to where I really want it.

I can do this for myself. I don't think I trust users to do it for themselves unless we know definitively that spam is configured on a per-mailbox basis (I don't think it is).

Any comments?

Jeff
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,793
Location
A Coruña, Spain
Yes, the Ham folder should be used for email marked as spam that are actually not spam.

The script doesnt delete/move emails on spam/ham folder once done, you can set it to delete the spam content changing

DELETE_SPAM_DATA=0
to
DELETE_SPAM_DATA=1

Regarding the level of learn, it does save bayes files in /home/USER/.spamassassin so most likely it doent learn per-mailbox but per-system-user basic

Regards
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
I don't feel comfortable setting it up, then, on shared servers, because account admin's (not server admins) don't have control over what their own users could cause to be marked as spam for all their users. What do you think?

I'm planning on moving my own email to my own server and if so I'll implement it for me.

Jeff
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,793
Location
A Coruña, Spain
Well if each domain have its own system user (ex. user: whatever domain:domain.tld) the learning method for the user whatever would just cause the settings for user whatever and it's own domains.

So, if a user set something wrong as spam, that change will be applied just for that user, not on all users on the system.

Regards
 

DirectAdmin Support

Administrator
Staff member
Joined
Feb 27, 2003
Messages
9,005
Martynas has added libspf2 into CB2, so when this is added, and exim recompiled, exim will have EXPERIMENTAL_SPF:
Code:
./build update
./build set easy_spam_fighter yes
./build exim
so that the Easy Spam Fighter 1.3 (I've just updated it on files1) can use exim's internal SPF checking, instead of using an external script.
It will also now drop SPF values that completely fail, and increase the spam score for softfail.

John
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
8,892
Location
LT, EU
I would like to add that "./build exim_conf" or "./build easy_spam_fighter" to update Easy Spam Fighter.
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,793
Location
A Coruña, Spain
Hi Martynas,

i was doing some few tests with those new options and i've tryed to set eximconf version to 4.3 (i'm actually using 4.3 so has been just a test).

Doin ./build exim_conf it work fine except for exim.conf file itself:

Code:
--2014-11-03 13:35:57--  http://files11.directadmin.com/services/SpamBlocker/4.3.0/exim.conf-SpamBlockerTechnology-v4.3.0.txt
Resolving files11.directadmin.com... 93.63.162.59
Connecting to files11.directadmin.com|93.63.162.59|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24504 (24K) [text/plain]
Saving to: `/etc/exim.conf.custombuild20'

100%[======================================================================================================================================================================================================>] 24,504      --.-K/s   in 0s

2014-11-03 13:36:03 (210 MB/s) - `/etc/exim.conf.custombuild20' saved [24504/24504]

Download of /etc/exim.conf failed
I've also noticed this diff with the one i'm using
Code:
177d176
< openssl_options = +no_sslv2 +no_sslv3
614c613
<   transport = virtual_localdelivery
---
>   transport = dovecot_lmtp_udp
This i suppose should be cause by pigeonhole, but i also suppose that since the file doesnt get renamed the patch (which i suppose should be a simple 'sed') doesnt get applyed, am i right?

Want me to give other output to check this behaviurs?

Not so hurry, i do have my working exim.conf save apart, just a notice :)

Best regards
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
8,892
Location
LT, EU
SeLLeRoNe, thank you for the report, please try CB 2.0 rev. 1100 (latest version of CB 2.0).
 

DirectAdmin Support

Administrator
Staff member
Joined
Feb 27, 2003
Messages
9,005
Just checked our test copy, which does seem to work in our setup.
Check /var/log/exim/mainlog and paniclog for anything obvious.

Of course, make sure the password is correct, and the send limit for this account or User is not currently hit.
The 550 should also include a message. What is it?

John
 

TomJones

Verified User
Joined
May 9, 2004
Messages
64
I'm using everything that's recommended (that I know of) on my DA server: Spamblocker 4.2, RBL, and SpamAssassin with training (I've given the trainer 500+ spam emails). We are still getting hammered, 7 out of 10 emails are spam, and one of my clients says they are getting 100+ every couple of hours. My understanding is 4.3 is not ready for production, but I am getting desperate. I'm an average admin, not advanced, but not a novice. I'm on a Centos 6, 64bit box using CustomBuild v.1. Can I use 4.3 in its current state?
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,793
Location
A Coruña, Spain
I'm already using SB 4.3 in production and is working pretty fine.

But i must say, SB 4.2 was working great and i wasnt getting so much spam as you are, so you would better check if you have some missocnfiguration somewhere.

Regards
 

TomJones

Verified User
Joined
May 9, 2004
Messages
64
My client that gets the most spam thinks it's more "personal" to send out monthly newsletters from their real email address, they list email addresses on their website in plain text, too. My company email address that gets hammered by most of the spam was listed as the registrar contact address for years; GoDaddy willfully vomited it up when anyone wanted to check out who owned our domain name. It's like a 30 to 1 ratio of spam to that address vs all the rest of my email addresses combined.

That said, spam seems to come in waves. The last few days were a storm of spam, then nothing today. I imagine they are newly compromised servers whose IPs haven't been blacklisted yet. I'm okay with a little spam, but getting nearly the exact same email about Russian Brides and prescription medications for cheap over & over starts to drive me nuts. I feed dozens & dozens of them to the SpamAssassin trainer (http://help.directadmin.com/item.php?id=358), but they just keep getting through. Spam is getting caught, because there is flagged spam in the account spam folder, it's just that for every 1 email caught, 10 get through. I set the threshold score in SpamAssassin to 2.0 (pretty strict), but the ones that get through are scoring from -0.0 to 1.0.

I think everything must be configured correctly; when I upgraded exim awhile back, I didn't use SpamBlocker at first and I was hammered by spam. Once I configured it again, the flow reduced dramatically. Just seems like the spammers are a step ahead of everything right now.
 

interfasys

Verified User
Joined
Oct 31, 2003
Messages
2,099
Location
Switzerland
Great news about the tokenisation of strings and variables.

Regarding Easy Spam Fighter, I would try to include OpenDmarc support. I would also make Spamassassin optional by commenting out those sections. The original script did not have any SA code in it.
ClamAV does a great job at identifying bad URLs in messages if you care to load some extra rules and there are good SURBL scanners out there.

Regarding teaching the antispam engine what's good and what's bad, the easiest method for both users and admins is to just monitor folders using Dovecot.

And finally, I'm also in favour of nolisting as opposed to using greylisting which unnecessarily delays emails.
 
Top