Mail Queue Administration list keeps becoming huge

apitsos

apitsos

Verified User
Joined
Dec 30, 2009
Messages
75
Location
Athens, Greece
Hi there,

Yesterday I had an issue with my web server. A client contacted me and he told me that his email were not delivered, even if in outlook they were in sent folder (IMAP connected). I tried to send a test message and I realized that he was right. Then I connected on DA control panel and I went to "Mail Queue Administration", where I saw 27 pages of emails in a queue list!

I tried to restart dovecot service, but I saw that the list were growing up instead of reducing! I rebooted the server and after reboot I tried to clear that queue list by checking the emails (page by page) and clicking "retry".

After an hour or something I cleared the list and everything seemed that was working fine.

Today afternoon I had again the same issues. I tried to find a solution but I couldn't I spent several hours, but nothing. Restarting the services of exim and dovecot couldn't give a solution. After several hours I make it to clear the list with retries.

What I notice is that the system produces decades of emails per minute! The strange is that the sender is empty and the recipient is the root user! I am placing here the header, the body and the log of one of these emails.

Header
Code: 
1TTyBc-0002vR-QC-H
mail 8 12
<>
1351789448 0
-ident mail
-received_protocol local
-body_linecount 161
-max_received_linelength 93
-allow_unqualified_recipient
-allow_unqualified_sender
-deliver_firsttime
-localerror
XX
1
[email protected]

146P Received: from mail by manage.pla.net.gr with local (Exim 4.72)
	id 1TTyBc-0002vR-QC
	for [email protected]; Thu, 01 Nov 2012 19:04:08 +0200
038  Date: Thu, 01 Nov 2012 19:04:08 +0200
050I Message-Id: <[email protected]>
044  X-Failed-Recipients: [email protected]
029  Auto-Submitted: auto-replied
061F From: Mail Delivery System <[email protected]>
027T To: [email protected]
059  Subject: Mail delivery failed: returning message to sender

Body Chunk
Code: 
1TTyBc-0002vR-QC-D
This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

  [email protected]
    retry timeout exceeded

------ This is a copy of the message, including all the headers. ------

Return-path: <[email protected]>
Received: from root by manage.pla.net.gr with local (Exim 4.72)
	(envelope-from <[email protected]>)
	id 1TTvlQ-0005l2-Pv
	for [email protected]; Thu, 01 Nov 2012 16:28:56 +0200
Date: Thu, 01 Nov 2012 16:28:56 +0200
Message-Id: <[email protected]>
To: [email protected]
Subject: lfd on manage.pla.net.gr: Suspicious process running under user govisit
From:  <[email protected]>

Time:    Thu Nov  1 16:28:56 2012 +0200
PID:     21006
Account: govisit
Uptime:  128 seconds


Executable:

/usr/local/php5/bin/php-cgi


Command Line (often faked in exploits):

/usr/local/php5/bin/php-cgi


Network connections by the process (if any):

tcp: 127.0.0.1:37508 -> 127.0.0.1:21


Files open by the process (if any):



Memory maps by the process (if any):

00110000-00183000 r-xp 00000000 fd:03 24035460   /usr/local/lib/libfreetype.so.6.7.1
00183000-00187000 rwxp 00073000 fd:03 24035460   /usr/local/lib/libfreetype.so.6.7.1
00187000-00188000 rwxp 00187000 00:00 0 
00188000-00192000 r-xp 00000000 fd:03 62489602   /lib/libnss_files-2.5.so
00192000-00193000 r-xp 00009000 fd:03 62489602   /lib/libnss_files-2.5.so
00193000-00194000 rwxp 0000a000 fd:03 62489602   /lib/libnss_files-2.5.so
00194000-00198000 r-xp 00000000 fd:03 62490542   /lib/libnss_dns-2.5.so
00198000-00199000 r-xp 00003000 fd:03 62490542   /lib/libnss_dns-2.5.so
00199000-0019a000 rwxp 00004000 fd:03 62490542   /lib/libnss_dns-2.5.so
0019a000-0019b000 rwxp 0019a000 00:00 0 
0019b000-001a3000 r-xp 00000000 fd:03 24028410   /usr/lib/libkrb5support.so.0.1
001a3000-001a4000 rwxp 00007000 fd:03 24028410   /usr/lib/libkrb5support.so.0.1
001a4000-001a7000 rwxp 001a4000 00:00 0 
001a7000-001a8000 r-xp 03425000 fd:03 24036482   /usr/lib/locale/locale-archive
001a8000-001a9000 r-xp 0085d000 fd:03 24036482   /usr/lib/locale/locale-archive
001a9000-001aa000 r-xp 02d67000 fd:03 24036482   /usr/lib/locale/locale-archive
001aa000-001ab000 r-xp 02edd000 fd:03 24036482   /usr/lib/locale/locale-archive
001af000-001b0000 rwxp 001af000 00:00 0 
001b0000-001ed000 r-xp 00000000 fd:03 24031561   /usr/local/lib/libpcre.so.0.0.1
001ed000-001ee000 rwxp 0003c000 fd:03 24031561   /usr/local/lib/libpcre.so.0.0.1
001ee000-00229000 r-xp 00000000 fd:03 62490523   /lib/libsepol.so.1
00229000-0022a000 rwxp 0003b000 fd:03 62490523   /lib/libsepol.so.1
0022a000-00234000 rwxp 0022a000 00:00 0 
00234000-0026f000 r-xp 01013000 fd:03 24036482   /usr/lib/locale/locale-archive
0026f000-00291000 r-xp 01be0000 fd:03 24036482   /usr/lib/locale/locale-archive
00291000-002ac000 r-xp 01cd9000 fd:03 24036482   /usr/lib/locale/locale-archive
002ac000-002bf000 r-xp 00000000 fd:03 24035472   /usr/local/lib/libz.so.1.2.3
002bf000-002c0000 rwxp 00012000 fd:03 24035472   /usr/local/lib/libz.so.1.2.3
002d3000-002d4000 rwxp 002d3000 00:00 0 
00323000-0036f000 r-xp 00000000 fd:03 24035459   /usr/local/lib/libcurl.so.4.2.0
0036f000-00371000 rwxp 0004b000 fd:03 24035459   /usr/local/lib/libcurl.so.4.2.0
003a8000-003a9000 r-xp 003a8000 00:00 0          [vdso]
003a9000-00463000 r-xp 00000000 fd:03 24641577   /usr/local/ioncube/ioncube_loader_lin_5.2.so
00463000-00468000 rwxp 000b9000 fd:03 24641577   /usr/local/ioncube/ioncube_loader_lin_5.2.so
00468000-00469000 rwxp 00468000 00:00 0 
00482000-004c6000 r-xp 00000000 fd:03 62489599   /lib/libssl.so.0.9.8e
004c6000-004ca000 rwxp 00043000 fd:03 62489599   /lib/libssl.so.0.9.8e
004ca000-0054b000 rwxp 004ca000 00:00 0 
0055c000-00577000 r-xp 00000000 fd:03 62488764   /lib/ld-2.5.so
00577000-00578000 r-xp 0001a000 fd:03 62488764   /lib/ld-2.5.so
00578000-00579000 rwxp 0001b000 fd:03 62488764   /lib/ld-2.5.so
005b1000-005b2000 rwxp 005b1000 00:00 0 
005b2000-006a7000 r-xp 00000000 fd:03 24035538   /usr/local/lib/libiconv.so.2.5.0
006a7000-006a8000 rwxp 000f5000 fd:03 24035538   /usr/local/lib/libiconv.so.2.5.0
006ba000-006bb000 rwxp 006ba000 00:00 0 
006d6000-006d9000 r-xp 00000000 fd:03 62488820   /lib/libdl-2.5.so
006d9000-006da000 r-xp 00002000 fd:03 62488820   /lib/libdl-2.5.so
006da000-006db000 rwxp 00003000 fd:03 62488820   /lib/libdl-2.5.so
006dd000-006f2000 r-xp 00000000 fd:03 62490528   /lib/libpthread-2.5.so
006f2000-006f3000 r-xp 00015000 fd:03 62490528   /lib/libpthread-2.5.so
006f3000-006f4000 rwxp 00016000 fd:03 62490528   /lib/libpthread-2.5.so
006f4000-006f6000 rwxp 006f4000 00:00 0 
006f8000-0071f000 r-xp 00000000 fd:03 62490526   /lib/libm-2.5.so
0071f000-00720000 r-xp 00026000 fd:03 62490526   /lib/libm-2.5.so
00720000-00721000 rwxp 00027000 fd:03 62490526   /lib/libm-2.5.so
00723000-0072a000 r-xp 00000000 fd:03 62490529   /lib/librt-2.5.so
0072a000-0072b000 r-xp 00007000 fd:03 62490529   /lib/librt-2.5.so
0072b000-0072c000 rwxp 00008000 fd:03 62490529   /lib/librt-2.5.so
0072e000-00744000 r-xp 00000000 fd:03 62490524   /lib/libselinux.so.1
00744000-00746000 rwxp 00015000 fd:03 62490524   /lib/libselinux.so.1
00790000-00799000 r-xp 00000000 fd:03 62490530   /lib/libcrypt-2.5.so
00799000-0079a000 r-xp 00008000 fd:03 62490530   /lib/libcrypt-2.5.so
0079a000-0079b000 rwxp 00009000 fd:03 62490530   /lib/libcrypt-2.5.so
0079b000-007c2000 rwxp 0079b000 00:00 0 
007c4000-007d9000 r-xp 00000000 fd:03 62490532   /lib/libnsl-2.5.so
007d9000-007da000 r-xp 00014000 fd:03 62490532   /lib/libnsl-2.5.so
007da000-007db000 rwxp 00015000 fd:03 62490532   /lib/libnsl-2.5.so
007db000-007dd000 rwxp 007db000 00:00 0 
0087f000-0088f000 r-xp 00000000 fd:03 62489477   /lib/libresolv-2.5.so
0088f000-00890000 r-xp 0000f000 fd:03 62489477   /lib/libresolv-2.5.so
00890000-00891000 rwxp 00010000 fd:03 62489477   /lib/libresolv-2.5.so
00891000-00893000 rwxp 00891000 00:00 0 
008c7000-008ee000 r-xp 00000000 fd:03 24035539   /usr/local/lib/libmcrypt.so.4.4.8
008ee000-008f1000 rwxp 00027000 fd:03 24035539   /usr/local/lib/libmcrypt.so.4.4.8
008f1000-008f6000 rwxp 008f1000 00:00 0 
009b2000-009f7000 r-xp 00000000 fd:03 24035671   /usr/local/lib/libmhash.so.2.0.1
009f7000-009f8000 rwxp 00044000 fd:03 24035671   /usr/local/lib/libmhash.so.2.0.1
00a41000-00a43000 r-xp 00000000 fd:03 62490525   /lib/libcom_err.so.2.1
00a43000-00a44000 rwxp 00001000 fd:03 62490525   /lib/libcom_err.so.2.1
00a46000-00a48000 r-xp 00000000 fd:03 62488901   /lib/libkeyutils-1.2.so
00a48000-00a49000 rwxp 00001000 fd:03 62488901   /lib/libkeyutils-1.2.so
00a8c000-00ab9000 r-xp 00000000 fd:03 24035672   /usr/lib/libgssapi_krb5.so.2.2
00ab9000-00aba000 rwxp 0002d000 fd:03 24035672   /usr/lib/libgssapi_krb5.so.2.2
00abc000-00b50000 r-xp 00000000 fd:03 24035669   /usr/lib/libkrb5.so.3.3
00b50000-00b53000 rwxp 00093000 fd:03 24035669   /usr/lib/libkrb5.so.3.3
00b55000-00b7b000 r-xp 00000000 fd:03 24031870   /usr/lib/libk5crypto.so.3.1
00b7b000-00b7c000 rwxp 00025000 fd:03 24031870   /usr/lib/libk5crypto.so.3.1
00b7c000-00ca6000 r-xp 00000000 fd:03 62489585   /lib/libcrypto.so.0.9.8e
00ca6000-00cb9000 rwxp 00129000 fd:03 62489585   /lib/libcrypto.so.0.9.8e
00cb9000-00cbd000 rwxp 00cb9000 00:00 0 
00d68000-00d6d000 r-xp 00000000 fd:03 24035675   /usr/local/lib/libltdl.so.3.1.0
00d6d000-00d6e000 rwxp 00004000 fd:03 24035675   /usr/local/lib/libltdl.so.3.1.0
00d6e000-00dbf000 r-xp 00000000 fd:03 24035390   /usr/lib/libmysqlclient.so.15.0.0
00dbf000-00eba000 rwxp 00050000 fd:03 24035390   /usr/lib/libmysqlclient.so.15.0.0
00f66000-00fa0000 r-xp 00000000 fd:03 24032163   /usr/local/lib/libpng.so.3.1.2.44
00fa0000-00fa1000 rwxp 0003a000 fd:03 24032163   /usr/local/lib/libpng.so.3.1.2.44
00fa1000-010f4000 r-xp 00000000 fd:03 62488804   /lib/libc-2.5.so
010f4000-010f6000 r-xp 00153000 fd:03 62488804   /lib/libc-2.5.so
010f6000-010f7000 rwxp 00155000 fd:03 62488804   /lib/libc-2.5.so
010f7000-010fa000 rwxp 010f7000 00:00 0 
0199c000-01ab5000 r-xp 00000000 fd:03 24035506   /usr/local/lib/libxml2.so.2.7.6
01ab5000-01aba000 rwxp 00118000 fd:03 24035506   /usr/local/lib/libxml2.so.2.7.6
01aba000-01abb000 rwxp 01aba000 00:00 0 
05c0a000-05d51000 r-xp 00000000 fd:03 24035548   /usr/local/lib/ZendOptimizer_5.2.so
05d51000-05d63000 rwxp 00146000 fd:03 24035548   /usr/local/lib/ZendOptimizer_5.2.so
05d63000-05d67000 rwxp 05d63000 00:00 0 
0637e000-0657e000 r-xp 00000000 fd:03 24036482   /usr/lib/locale/locale-archive
08048000-08602000 r-xp 00000000 fd:03 24742722   /usr/local/php5/bin/php-cgi
08602000-0862d000 rwxp 005ba000 fd:03 24742722   /usr/local/php5/bin/php-cgi
0862d000-08637000 rwxp 0862d000 00:00 0 
088b9000-09369000 rwxp 088b9000 00:00 0          [heap]
bff17000-bff2f000 rwxp bffe6000 00:00 0          [stack]

Log
Code: 
2012-11-01 19:04:08 Received from <> R=1TTvlQ-0005l2-Pv U=mail P=local S=9306 T="Mail delivery failed: returning message to sender"


I would appreciate if someone could help me with it! I strongly believe that something is wrong with my system the last two days and I need your help!


Kind regards,
Angelos Pitsos
 
The email you listed in the example above is sent from csf on your server to your server root email, it is not able to be delivered because you have not configured email for that root email address. You can solve that problem by adding a forwarder for the root email address. Do this:

Create a file wich has the file name: .forward (it shold not have any other extensions)

In the .forward file you type one of your own email address that you use regulary, and nothing else, only content of the file should be a working email address.

Then upload .forward file to /root folder on your server.

This will solve the problem in the email you have in your example above, so that any email too server root email is delivered and will not stay in mail queue, or/and will not bounce back to the root email that is not working.

I do not know if you have any other problems that must be fixed or not, but this is one thing to do first.
 
Last edited:
Hi ditto!

Thank you so much for your reply. I will try that immediately, but I have two questions.

You said that "...csf on your server to your server root email...". The question is why and I mean why now? All this time, for months, I had no problems at all. And I do not understand what is trying csf to send to the root user?

The second question is what happened to those emails if there is no root email address. Do they deleted automatically after a while? And if yes, then why the list become huge and the mail server didn't deliver anything at the end? Because it happened the mail server not to deliver emails for several hours. That's how I realized that something was wrong!

Thanks again for your time and your help.


Regards,
Angelos Pitsos
 
Question 1: You seem to have installed csf on your server (http://configserver.com/cp/csf.html), and csf is sending email to your root email every time it has some alert message for you. You should configure csf in your server to fit your needs.

Question 2: It seems that csf sent many email alerts to your root email wich was not working, and the sender of those emails was also the server root email, so csf send emails alerts from root email to root email, it is not able to be delivered and then it bounce back to root email wich ALSO is not able to be delivered, and the problem continue growing like this.
 
I see... So maybe there is more traffic on my server or more trials for hack and that's why csf is sending more alerts than usual... Am I correct?

Also what happened to those emails? Are they deleted automatically after some hour(s). And is this storm of emails possible to make mail server irresponsible or at least unable to deliver the emails that are in queue list?

Please note that I created that file and I have already blocked on of my emails! Outlook does not respond!!!!!!

Thanks again!


Angelos
 
Some alerts csf is sending can be ignored, other alerts csf might send could be serious issues wich you need to inspect.

The emails sent to your server root email will be deleted from your server if you have configured your email client to delete the messages from the server when you receive them in your email client.

Outlook is probably not responding because there is too many emails sent to root user. You should make sure you receive all so they are deleted, or you should delete all from mail queue in DirectAdmin.

It seems to me you need a server administrator to help you manage your servers, so maybe you should pay someone for help.
 
ditto,

Are you able to undertake this small project and solve it, please? Are you a linux system administrator? and if yes, do you have an account on freelancer.com, as I posted already a project for this purpose.


Kind regards,
Angelos Pitsos
 
I don't provide server administration, however I can personally recommend SeLLeRoNe from this forum, because I payed him to help me one time, and he did a great job. There is also several other server administrators in this forum wich has good reputation and offer server administration. Here is some one them:

SeLLeRoNe http://www.directadmin.com/forum/member.php?u=1773

zEitEr http://www.directadmin.com/forum/member.php?u=2806

nobaloney http://www.directadmin.com/forum/member.php?u=48

smtalk http://www.directadmin.com/forum/member.php?u=5982
 
Yes, Jeff seems to be a very good choice. I listed him in above reply with user name "nobaloney". Also, if you are in a hurry, you might want to send a pm to zEitEr, because I see on his profile page that he is online in the forum now. Good luck!
 
Thank you so much ditto! You input is much appreciated and you helped me.

I know that Jeff is the "nobaloney" user ;-)

Regards,
Angel
 
The strange thing with all this is that the mail server is stuck again! I setup the forwarder as suggested. It sent about 15 emails and now I see the queue list to growing up again :(
 
Well, my replies was only based on the ONE email you listed in your example above. Maybe one of your users is "hacked" and is sending out big amounts of spam? I can't be of any more help. I hope you soon get in contact with someone who can help you.
 
apitsos said:
No problem. Thank you so much...
Click to expand...
I had a lot of problems when I first installed CSF but I finally found a configuration that works for me without creating all those emails from warnings I don't want or need. Now I maintain one copy of the /etc/csf directory, and after installing CSF on a server, I upload my copy for a fast configuration. I can't send it to you because it's customized for my company.

To answer some of your questions in this thread:

Dovecot has nothing to do with the queue; keep that in mind for the future; save a step :).

Your exim.conf file settings should automatically clear messages off the queue after some time, but the time may need to be shortened to keep your queue from growing.

Make sure that you're forwarding all emails from CSF to go to an unlimited data mailbox, and read them (or at least scan them) to see if something is now attacking your server; the whole purpose of the emails is to inform you :).

Jeff
 
I had a kind of similar problem in the past to and solved it this way.
1.) In the /etc/aliases file, I put my email address like this at the bottom:
Code: 
root:     [email protected]
Then I went to /etc/virtual and checked if my hostname was there, in your case it would be a nice example:
Code: 
drwx--x--x 2 mail      mail   4.0K Jul 18 18:48 manage.pla.net.gr

Then restart Exim and see if the problem is over.
 
Dear all,

I would like to thank you very much for your replies. The problem is solved and actually it was not related with the mail server at all.

The problem was the high load that the server had the last two and a half days. What caused that high load? It was coming from a website that was hosted under the account of one of my clients. It's about a joomla site and it was hacked. For this reason I had to suspend the specific account, and suddenly, everything became normal!

You live and learn...! But I am starting to hate that kind of CMS, called Joomla! I am using it and I loved Joomla, but the last weeks I have realized that is the worst solution for web development. Apart from the easy installation, configuration and development, there is nothing else that could make it a strong player. On the other hand, I have seen other kind of open source CMS platforms that are exceptional.

Anyway, everything is fine and the server is now stable, apart from the suspended account of my client...!


Thanks all of you for your time.


kind regards,
Angelos Pitsos
 
I also have a problem with my Mail Queue growing, one one of my servers. For some unexplained reason, recently this mail queue always contains legitimate emails that when 'retried' manually, instantly get delivered but for some reason are not processed automatically. The emails that are in the queue just sit there for hours and days, not being processed automatically. After discovering this, there were 12 pages of emails in queue, now it is generally limited to one page every day

It consists of SOME of the total number of sent/received emails, many emails do send and receive immediately. This includes both incoming and outgoing emails, most of which are valid, legitimate emails (not spam, dodgy to/from, etc.).

Emails are server status emails (sent to and from various servers) as well as 'hand written' client emails (several email accounts under one user-account). The server has a very low load, plenty of diskspace and never had email issues before.

Any thoughts on cause, solution or work-around are welcome!

Kind regards,

Harro
 
Hi harro!

Have you checked you Joomla installations? As I said on my last post, the problem was caused actually by a hack, where someone did hack a Joomla account and he was importing thousands of records per day on one of the tables of this Joomla installation.

Know I have a great Linux Administrator from Lebanon, who is helping me ad-hoc when I have issues with servers. That makes me sleep a little easier... ;-) i could recommend him if you are interested.


Kind regards,
Angelos Pitsos
 
Thank you for your feedback Angelos.

There is no Joomla (or other popular CMS) on the server and the server load is minimal, between 1 - 2 in the 'top' output. So, I don't feel that the server is too busy to process the mail queue. Also, most emails pass through / are processed successfully, just a (seemingly random) number are trapped in the queue. Weird, right?

Is there a log file somewhere that logs the mail queue decisions (whether to send, freeze, delay, retry, emails in the queue)? Maybe such a file could provide an insight in why certain emails get stuck...?
 
You must log in or register to reply here.
Back
Top