Shell script for automating secondary MX

[daz]

Hi Guys,

Ok, I have been hangin around picking up tips and getting help and now it is time to give something back

I needed to set up a secondary mx server for several hosting servers, some of which run DA. I will try and make this as foolproof as possible, but it may vary from distro to distro.

# First off, lets create a user on the backupmx server
$ useradd backupmx -d /home/backupmx -m -s /bin/bash
# Now create an ssh key for the new user
$ su backupmx
$ ssh-keygen -t rsa
# Grab the ssh key and stick it in notepad for future reference
$ cat ~/.ssh/id_rsa.pub

# Now, on each of the hosting servers:

# create user for the backupmx scripts to run
$ useradd backupmx -d /home/backupmx -m -s /bin/bash
$ su backupmx
$ nano ~/.ssh/authorized_keys
# put the key that you copied from the backup server in here

## NOTE ##
If there is no .ssh dir in the home directory of the user you are working with (there probably wont be if it is a new user), just ssh somewhere and it will be automatically created.

On my system, exim configs reside in /etc/exim4, feel free to adjust if yours is different.

# Back to the backupmx server
# Create dir structure
$mkdir /etc/exim4/backupmx
$mkdir /etc/exim4/backupmx/domain_files
$ chown -R backupmx:backupmx /etc/exim4/backupmx

And then copy this script to /etc/exim4/backupmx/retrieve_domains.sh

# Set Variables
cd /etc/exim4/backupmx/

# Import from servers

# SERVER1
ssh [email protected] cat /etc/virtual/domains > ./domain_files/server1

# SERVER2
ssh [email protected] cat /etc/virtual/domains > ./domain_files/server2

# Insert into exim4

# initialise
if [ -e domains ] ; then
  rm domains
fi
if [ -e domains.temp ] ; then
  rm domains.temp
fi
touch domains

for file in $( ls ./domain_files/ )
do
  cat ./domain_files/$file >> domains.temp
done

# remove non-domain lines
sed -i '/^([a-zA-Z0-9]([a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,6}$/d' domains.temp

# remove duplicates to keep it tidy
sort domains.temp | uniq > domains

# clean up and exit normally
rm domains.temp
exit 0

Now, edit your exim.conf and set the following line:

domainlist relay_to_domains = <stuff that you already have> : lsearch;/etc/exim4/backupmx/domains

# Reload exim..
$ /etc/init.d/exim4 restart

Finally, set up a cron job for the script

5 * * * * /etc/exim4/backupmx/retrieve_domains.sh > /dev/null

(runs on the 5th minute of every hour - adjust to suit)

And now for the magic...

Oh wait, there is nothing left to do!

Except set the dns records of course...

You can test if the system works by running this command on the backupmx server:

$ exim -v [email protected]
test message
LOG: MAIN
  <= [email protected] U=root P=local S=361
root@backupmx:~# delivering 1JyKtS-0001hj-3l
R: dnslookup_relay_to_domains for [email protected]
T: remote_smtp for [email protected]
Connecting to mail.relaydomain.com [2.2.2.2]:25 ... connected
  SMTP<< 220 server2.relaydomain.com ESMTP Exim 4.63 Tue, 20 May 2008 15:43:50 +1000
  SMTP>> EHLO backupmx.mydomain.com
  SMTP<< 250-server2.relaydomain.com Hello backupmx.mydomain.com [3.3.3.3]
         250-SIZE 20971520
         250-PIPELINING
         250-AUTH PLAIN LOGIN
         250-STARTTLS
         250 HELP
  SMTP>> STARTTLS
  SMTP<< 454 TLS currently unavailable
  SMTP>> MAIL FROM:<[email protected]>
  SMTP<< 250 OK
  SMTP>> RCPT TO:<[email protected]>
  SMTP<< 250 Accepted
  SMTP>> DATA
  SMTP<< 354 Enter message, ending with "." on a line by itself
  SMTP>> writing message and terminating "."
  SMTP<< 250 OK id=1JyKda-0005zQ-MK
  SMTP>> QUIT
LOG: MAIN
  => [email protected] R=dnslookup_relay_to_domains T=remote_smtp H=mail.relaydomain.com [2.2.2.2]
LOG: MAIN
  Completed

Obviously, if you have non-DA servers that you want to run a backupmx for, you can do that simply by changing the ssh line for that server to point to /etc/localdomains or where ever the list of domains is kept.

I hope this helps someone

 

[nobaloney]

I haven't tried your script.

But I don't see any reference to the domainlist relay_to_domains other than the definition.

By default exim doesn't do anything special with a domainlist.

Where have you defined the usage of the domainlist? Is that something in your exim.conf file that you've forgotten to post?

Jeff

 

[daz]

Hi Jeff,

I used the standard config that came with ubuntu 8.04 (after running the exim config tool of course). It is something I overlooked, as I assumed people would be setting up the backup mx server from a fresh install, where as people with an existing exim conf will need to add rules for that domainlist if they do not already exist.

I believe adding this block to the end of your acl/30_exim4-config_check_rcpt section should allow exim to relay, but not being an exim expert I would welcome corrections

  accept
    domains = +relay_to_domains
    endpass
    verify = recipient

I checked the config on my DA servers as well, and if you are running your backup mx on a DA server it should be fine just to add to the domainlist (as per http://help.directadmin.com/item.php?id=167).

 

[nobaloney]

Our exim.conf files don't have an acl/30_exim4-config_check_rcpt either; this kind of makes me wonder if you've ever even looked at a DirectAdmin exim.conf file.

For your information, and for others; in our exim.conf file, the corresponding section would be the check_recipient section, almost at the bottom of that section.

Using the SpamBlocker Version 3.1-beta of exim.conf, you'd put those lines directly above this line:

# FINAL DENY EMAIL BEFORE DATA BEGINS HERE

.

Once you're using DirectAdmin as a failover MX server queue runners become important; you should probably make sure you understand all the settings you can find by searching exim.conf for the word queue. Also, perhaps especially, important are ignore_bounce_errors_after and timeout_frozen_after as these settings control how long exim will hold undeliverable email in it's queue before freezing it as undeliverable.

Which brings up another important point.

What happens in this scenario:

1) you relay mail for example.com when the main mx server is down

2) somone sends an email to [email protected]

3) your failover mx server accepts the email, becuse the main mx server for example.com is down.

4) the main mx server for example.com comes back up and your server tries to relay the email to [email protected], but [email protected] is NOT a valid user at the main mx server.

Does your server attempt to return the email, and if so, to where? Or does it just drop it?

If the former, and if the email is spam, then it's likely your server is creating collateral spam; sending spam (as a return) to a forged address which never sent it.

If the latter, and the email is NOT spam, then it's likely your server is dropping email because the original user meant to send the email to [email protected] and accidentally sent it to [email protected], and the sender never got the email back, so he doesn't know his email didn't get through.

Which is it?

Thanks.

Jeff

 

[daz]

Our exim.conf files don't have an acl/30_exim4-config_check_rcpt either; this kind of makes me wonder if you've ever even looked at a DirectAdmin exim.conf file.

I built my backup mx server as a standalone server (ie: an offnetwork vm), hence my exim conf does not resemble the DA exim conf at all. The idea of a backupmx is to receive mail in the event of the main mail server being uncontactable (which includes network outages and server down issues) hence it did not make sense for me to put this on a production hosting server.

For your information, and for others; in our exim.conf file, the corresponding section would be the check_recipient section, almost at the bottom of that section.

Using the SpamBlocker Version 3.1-beta of exim.conf, you'd put those lines directly above this line:

# FINAL DENY EMAIL BEFORE DATA BEGINS HERE

.

Thank you for clearing that up. Again, I chose to use a new ubuntu install for my backup server, however I am certain that there will be others who wish to use a DA server and it is good to have all the instructions in the same place.

Which brings up another important point.

What happens in this scenario:

1) you relay mail for example.com when the main mx server is down

2) somone sends an email to [email protected]

3) your failover mx server accepts the email, becuse the main mx server for example.com is down.

4) the main mx server for example.com comes back up and your server tries to relay the email to [email protected], but [email protected] is NOT a valid user at the main mx server.

Does your server attempt to return the email, and if so, to where? Or does it just drop it?

If the former, and if the email is spam, then it's likely your server is creating collateral spam; sending spam (as a return) to a forged address which never sent it.

If the latter, and the email is NOT spam, then it's likely your server is dropping email because the original user meant to send the email to [email protected] and accidentally sent it to [email protected], and the sender never got the email back, so he doesn't know his email didn't get through.

Which is it?

Thanks.

Jeff

In that instance a bounce message is returned to the sender informing them that their email did not go through.

I accept that this creates spam, however all I have done is set up a basic backupmx and tried to help others do the same thing. Obviously a more complex setup is required (ie: spam filtering on the backupmx) but most people will elect (as I did) to set up a basic "get it working" system before adding the bells and whistles.

All I wanted to do was share a simple script I had written that collects all of the domains from your hosting servers and collates them into one file for easy insertion into exim. I am not strong on exim configs and generally don't play inside the config of a control panel. If you have a better howto for the exim side of this, please post it, otherwise remember that the thread title is "Shell script for automating secondary MX" not "how to set up exim for backup mx". The howto was only added so that there was some kind of explanation of how to use it for people who were not sure.

 

[nobaloney]

I accept that this creates spam,

That appears to be a rather cavalier attitude. Your server is probably in violation of terms of service at most upstream providers; I know it is with us.

however all I have done is set up a basic backupmx and tried to help others do the same thing. Obviously a more complex setup is required (ie: spam filtering on the backupmx) but most people will elect (as I did) to set up a basic "get it working" system before adding the bells and whistles.

The real problem is that your basic method of accepting email on your failover mx serveris flawed.

All I wanted to do was share a simple script I had written that collects all of the domains from your hosting servers and collates them into one file for easy insertion into exim.

Getting the domains isn't enough. What you need to get is the complete list of usernames for whom to accept email.

I am not strong on exim configs and generally don't play inside the config of a control panel. If you have a better howto for the exim side of this, please post it, otherwise remember that the thread title is "Shell script for automating secondary MX" not "how to set up exim for backup mx". The howto was only added so that there was some kind of explanation of how to use it for people who were not sure.

My How-To was to not do failover mx at all; I've made that clear in many previous posts and replies. I was hoping you'd come up with a method for doing failover mx that would work.

I hope you're not using your failover server now, and that you won't use it until you've resolved the collateral spam problem. Which is real. And will get you on block lists, and may put you in violation of terms of service that end up costing you and your upstream time, aggravation and money.

I suppose you might ask why I believe failover mx is unnecessary. Here's why:

Mail delivery includes a failover mechanism now.

Must mailservers will attempt to deliver mail for up to four days before giving up and returning email to the actual sender for whom it is actually responsible. Any collateral spam is going back to the spammer's actual mailserver (the one contributing to the spam, and not your innocent bystander).

Unless you believe that failover should extend beyond four days without notifying the sender that there's been a delay in the email, there's zero benefit from a failover mailserver.

That said, right now the default SpamBlocker version 3.1-beta exim.conf file only waits two days; that was done to keep the queue from getting too big on most servers. It's a compromise which will be resolved in the final version.

I have to recommend that anyone willing to set up an mx server following your suggestions be sure that their upstream provider is willing to accept being a sender of collateral spam.

Jeff