Receive email with exim, send it with something else

[koudou]

Hello to all of you,

I have a lot of problems beginning with exim and greylisting.

This is mainly due to electronic magazines (quite heavy) being sent on my server through exim (4.60) to emails on servers where they apply different policies of greylisting.

This means that my email is put in temporary error, and the email go to the queue of exim.

I dig on the net for informations on how exim is managing the queue, and my conclusion is that the answer is "not very well". The final result is that today, It takes 6 hours for exim to have 2000 messages going out from the queue (I have something like 15000 in the queue) with two queue runners.
Each queue runner is not able to begin to manage another message until the message they are trying to send is accepted with proper acknowledge or reach time out (at least 60s if I am taking a risk or 120s if I am traditional)

My thought is that I have only two options :
-> write an exim queue manager where I put more queue runners in parrallel
-> use another server with a more efficient MTA than Exim to take care of large queues, but only to send emails (I can plan to have a server only to relay)

My questions :
- Is there anybody who had the same problem and if some, have you some other suggestions ?
- how can I manage the process of sending emails with something else than exim. The best process should let the message being handled by exim, but, as soon that it comes inside exim, to reroute it to another server to handle it (another IP, etc... kind of immediate relay).
- Have you suggestion on which Unix systems using something else than exim (Postfix ?) I could use on this other server ?

Note : I want to keep exim on main server as it is integrated inside DirectAdmin email management and only delegate the outgoing of emails to another server)

Thank you in advance, and I think that this problem will be more and more important as a lot of servers are implementing today the greylisting system pushing the exim queue to high high loads.

Michel Lo

 

[nobaloney]

You can certainly install another server for outgoing emails; just make sure it doesn't listen to the outside world on port 25, as only one mailserver can bind to port 25 at a time.

You can even write your own .

But you can certainly manage exim queues; search for "queue" in the exim.conf file and you'll find a lot of settings you can change.

It all depends on how much memory you have.

Jeff

 

[koudou]

Hi Jeff,

I had carefully checked on queue configuration inside exim.conf. As I am using the one you have done for DA, there are not so many options left inside (I mean you probably took out all the unnecessary stuff : I am using your SpamBlocker.exim.conf.2.0-release)

The only things I did was to do a
queue_only=false
To try to deliver the message and not put it into the queue at first arrival

and
queue_run_max=20 to try to have up to 20 queue runners at a time.

But in fact, it appears that I still have two queue runners on, and only two messages are processed from the queue at the same time. And, as the queue runners are not able to handle any other message until it has finished with the message they are processing EVEN IF THEY ARE DOING NOTHING SAVE WAITING, 15000 messages takes more than 3 days to be delivered.

=============================
The whole problem comes from there : some newsletters are sent in large quantity (they have many opt-in subscribers and they are more and more coming) up to 30000. As many of these subscribers are in community organizations, their servers are using more and more greylist systems. And at first attempt, the email is not accepted (identification of message based on IP+sender+message-ID). And the message goes to queue.
As the emails are easy to greylist by sending back a temporary error, all the 30000 magazines are going very fast into the queue.
Some of the servers using a greylist demand that the message comes three times to be accepted...

And the messages are not going out from the queue fast enough.
============

I understand the suggestion to write my own email and the blue emoticon
going with it. But the solution of having a specialized server able to handle the queue sounbds to be the best solution as I suspect that processing queue will be consuming in the next future.

The only thing is that I do not know exactly how exim send its messages in the queue. I suppose that of course, it is in the code, and based on parameters. But I haven't identified if it is possible to parameter exim to send all out-going messages to another server.


Michel

 

[nobaloney]

There are a few more settings available in the latest beta version, which is what we now recommend. (Soon to be released.)

You don't write which program you're using to send the emails. Some commercial ones include their own server for sending the email; others (including some of the free ones) allow you to set a maximum amount of emails per minute.

We have one client who uses a commercial program which limits his outgoing emails to one per second; in ten hours he sends 36,000 emails; in one day (24 hours) he sends 86,400. His server has never had an overload.

Jeff

 

[koudou]

I am using sympa as the tool to manage email sending. (see
http://sympa.org : it is mainly in english)

In fact sympa is free as it has been developped by university community (french one), but today, it is international. We have developped a whole platform (mainly with a company I am shareholder in Malaysia) allowing to manage all the contents (for website, and building the newsletter with one click in three versions : text, html and pdf). Users can manage their subscription with sympa (public screens) and all sending management is done by sympa (bck office for bounce, retries, etc...), except for sending the mail itself. Sympa pass the mail to the MTA of the server and do not manage the level of sending process.

Of course, I can slower the process of sympa by sympa configuration. But this does not changes the fact that more and more servers are answering a temporary error with any new set of [IP, message id, sender]. This has for effect to put the message in the queue of exim.
I have for the moment three customer sending newsletters on one server (one full server for only 3 customers !). If I am not lucky, they send on the same evening 120 000 newsletters. At one per second, it will take 33 hours to send, and every 15 minutes (the time before I request exim to try to resend), I would have 600 new messages going into the queue (I cannot put faster than 10/15 mn for retries as some greylist systems are tuned to forbid answers coming before a certain amount of time, very often 2 minutes sometimes longer time).

Sometimes, it takes up to one minute to process a message by a queue runner of exim as it waits until the last acknowledgement from the destination server and do not begin the process of the next message before getting this ack.
Imagine the equation :
-> one message per minute an by runner
-> two runners at the same time
-> 60 new messages going in the queue and waiting to be processed every minute
-> 33 hours in a row of this situation

For the moment, I can get along because they do not send in the same time, but if this happens, it can become a huge problem (I have daily newspaper prospects).

Michel

 

[nobaloney]

The link you send is useless unless someone already knows what the project is and wants to know how to download it. There's no explanation at all.

Are you asking a question? I don't see one.

Are you pointing out that 120,000 newsletters are being sent through your exim server at once and that your exim installation has a problem with that? Of course it will.

You can adjust how and when exim calls queues, and even how many it will run at one time. Then your problem will more server load and crash.

If your mailing program doesn't know how to limit what it sends, then the problem is with the mailer.

Our exim.conf file is not designed for mass mailing; it's designed to work in a shared webhosting environment.

You can rewrite the exim.conf file if you wish; that's the beauty of open source .

Or you can find settings in Sympa to limit the emails to someting that the current exim configuration can handle.

Jeff

 

[koudou]

Apparently, too many explanations is too much.

My question is : how to tell to exim not to put an email in the queue in case of error given by the destination, but to forward it to another server.


Michel Lo

 

[nobaloney]

I don't know if it's possible.

Perhaps a good question for the exim-users mailing list?

Jeff