Another email queue question

nealdxmhost

Verified User
Joined
Jan 1, 2009
Messages
237
Location
Los Angeles CA
I have a client who has a large mailing list for a newsletter that he sends out periodically and the list is rather sizable (7000+), anyhow the site is built with Joomla and a newsletter component has been installed for him to send out the articles.

Anyhow the issue arises when he proceeds to send out the newsletter to that many subscribers all at once and what ends up happening is that I end up with an enormously large mail queue that takes a considerable amount of time to process. This is in turn has resulted in other clients calling me and complaining that they are having trouble with their emails (usually accessing them or sending them). When this has happened I have usually had to clear the queue of messages attributable to that one user and then email starts working again.

Anyhow I have advised the client to break that one large list down into considerably smaller sized lists (500 addresses) and work the lists one at a time, in other words sending the newsletter to one list at a time to minimize congestion. Additionally I advised the client to do their email blasts during off peak hours

Does this work? Trying to keep the mail server working the right way and at the same time meet the clients needs as well (additionally the client has opt-in and opt-out options for the newsletter). I had assisted them about two weeks ago with setting up majordomo to handle which appeared to work however they could not get any statistics as to the results of the emails being sent out.

I am trying to keep things from going nutty on my server again since re-worked everything on the server last week by doing a
Code:
 ./build all d
in custombuild

Slowly but surely I am trying to get a grip on all of this and I try to remember a certain mantra that friend told me a couple years back;
Code:
[SIZE="5"][COLOR="Red"][B]Open Source may be free but it sure ain't cheap![/B][/COLOR][/SIZE]
 
Hi


Linux use sendmail program for email delivery,

you shall replace sendmail with a custom script that will log

every email that being sent to a logfile in /var/log/spam with the user who run sent the email and from which directory the code has been run.


i can enhance the code so it will limit a user for 200msgs/day , i belive it can be a good solution.


to install the script
just rename the file /usr/sbin/sendmail to /usr/sbin/sendmail.hidden

and create a new sendmail file in the same directory
cut & paste this code


#!/usr//bin/perl

# use strict;
use Env;
my $date = `date`;
chomp $date;
open (INFO, ">>/var/log/spam_log") || die "Failed to open file ::$!";
my $uid = $>;
my @info = getpwuid($uid);
if($REMOTE_ADDR) {
print INFO "$date - $REMOTE_ADDR ran $SCRIPT_NAME at $SERVER_NAME n";
}
else {

print INFO "$date - $PWD - @infon";

}
my $mailprog = '/usr/sbin/sendmail.hidden';
foreach (@ARGV) {
$arg="$arg" . " $_";
}

open (MAIL,"|$mailprog $arg") || die "cannot open $mailprog: $!n";
while (<STDIN> ) {
print MAIL;
}
close (INFO);
close (MAIL);

and make sure you chmod 755 the file
 
He is using DirectAdmin, so Exim is used and not only sendmail. If you configure your mailing correctly it should use SMTP authentication with a valid emailaddress.

ontopic;
We tell our clients too if they like to send a mailing that they should use a bash of 500 mails, when the 500 mails are send, start the new bach.

One of our clients made a script, so it drops all mails in MySQL, then wrote a script it will send XXX mails, after that removing the mails from the table, and then start again.. until its empty.
 
@Neal:

Good (and sometimes expensive) mailing list programs can limit outgoing emails themseles.

That said, you can't determine how to fix the problem until you understand the problem. Your first step would be to check your logs to see why the outgoing emails are even ending up in the queue; they shouldn't be.

Jeff
 
Take a look at Exim optimizations eventually; We run queues with above 20000 outgoing mails (mostly mailings) but don't have any problems with delivering other mail to our users (locally or external).

Things that worked for us:
  • Write our mail software in such way it delivers 100 mails per SMTP connection to localhost, because if you deliver more than 10 messages in one SMTP connection, Exim queues the mail for slower/later delivery, and it doesn't congest other (important) mail.
  • Using split_spool_directory to help Exim handle large queues.
  • Setting things like "queue_run_max = 20".
  • Write our mailscript in such way it delivers only a few hundred messages per minute to Exim, letting the queue gradually build.

Hope that helps!

(P.S. For delivering multiple messages in one SMTP-connection; take a look at PHPMailer, it can open a SMTP connection once, and send loads of messages through it, before closing it)
 
Last edited:
@Neal:

Good (and sometimes expensive) mailing list programs can limit outgoing emails themseles.

That said, you can't determine how to fix the problem until you understand the problem. Your first step would be to check your logs to see why the outgoing emails are even ending up in the queue; they shouldn't be.

Jeff

Good point Jeff and that is one of the things that I am trying my level best to grasp the concept of, however something is slipping by me.
 
Take a look at Exim optimizations eventually; We run queues with above 20000 outgoing mails (mostly mailings) but don't have any problems with delivering other mail to our users (locally or external).

Things that worked for us:
  • Write our mail software in such way it delivers 100 mails per SMTP connection to localhost, because if you deliver more than 10 messages in one SMTP connection, Exim queues the mail for slower/later delivery, and it doesn't congest other (important) mail.
  • Using split_spool_directory to help Exim handle large queues.
  • Setting things like "queue_run_max = 20".
  • Write our mailscript in such way it delivers only a few hundred messages per minute to Exim, letting the queue gradually build.

Hope that helps!

(P.S. For delivering multiple messages in one SMTP-connection; take a look at PHPMailer, it can open a SMTP connection once, and send loads of messages through it, before closing it)

Could you expand a little bit on the "split_spool_directory" and the "queue_run_max" a bit for me?
 
@AndriesLouw:

I can't find any limit of 10 messages per SMTP connection in either my newest exim.conf file for DirectAdmin. Please tell me which directive sets this.

From my log I get messages like:
Code:
2010-10-12 07:31:10 1P5XSA-0005ck-6P no immediate delivery: more than 10 messages received in one connection
I believe this is default behavior from Exim, and I like it ;)

Edit: Found more information about it on http://www.exim.org/exim-html-4.50/doc/html/FAQ_0.html -> Search for "more than 10 messages received" or "Q0049"

Could you expand a little bit on the "split_spool_directory" and the "queue_run_max" a bit for me?

These options split the spool directory so that Exim (especially on Linux) can handle the larger spool files better, as well as create multiple spool threads (in our case, 20 threads will simultaneously deliver mail). To read more about this directives you can consult various sites with information about Exim-tuning, or look it up in the Exim-manual (search a few times for split_spool_directory or queue_run_max on that page).
 
Last edited:
I'll probably go ahead and add the default of:
smtp_accept_queue_per_connection=10 to the new file the next time I make a change, just to make it obvious and easier to change.

But I'm not going to make changes to the spool system to add multiple directories, or add more queue runs by default; I believe from the lack of issues posted on these forums that the standards are working for the majority of us. They certainly work for me, and in our case we never get more than a few hundred emails in the queue at most except when users are attempting to spam.

Jeff
 
Back
Top