Server unable to send emails at all - stuck in queue - SMTP timeout after initial connection

baggs1981

Verified User
Joined
Sep 18, 2006
Messages
66
Hi All, I have a server which suddenly stopped being able to send emails. All sent emails from any domain on the server go into the queue and never process. This is even when sending from the server itself such as roundcube webmail.

Every sent email goes straight into the queue, has error like this and never delivers:

Code:
2024-03-31 21:08:18 Received from [email protected] H=(smtpclient.apple) [90.248.26.137] P=esmtpa A=plain:[email protected] S=9146 [email protected] T="Re: May. annual review reminder"
2024-03-31 21:13:18 H=alt1.gmail-smtp-in.l.google.com [142.250.153.27]: SMTP timeout after initial connection: Connection timed out
2024-03-31 21:13:18 [email protected] R=lookuphost T=remote_smtp defer (110): Connection timed out H=alt1.gmail-smtp-in.l.google.com [142.250.153.27]: SMTP timeout after initial connection

SSl check:
openssl s_client -connect server01.domain.net:25 -starttls smtp -servername server01.domain.net

Code:
-----END CERTIFICATE-----
subject=/CN=server01.domain.net
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2950 bytes and written 485 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-ECDSA-AES128-GCM-SHA256
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES128-GCM-SHA256
    Session-ID: DA53788801EBA6F4827BE46502CD64E3EC5A391540FDE03F12FB72A2C49EBBC6
    Session-ID-ctx:
    Master-Key: AC333ECD891FC083339A0479AFDw4D51F3C72E28e4203D10C65d5FA70ECF489AED3C81CD41F140F2E1F8A54D3B79BFC2
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1711916082
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
250 HELP

Exim Check:
exim -d -bh 127.0.0.1

Code:
Exim version 4.97.1 uid=0 gid=0 pid=483 D=f7715cfd
Support for: Content_Scanning crypteq iconv() IPv6 Perl move_frozen_messages OpenSSL TLS_resume DANE DKIM DNSSEC Event OCSP PIPECONNECT PRDR Queue_Ramp SPF SRS TCP_Fast_Open
Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb
Authenticators: cram_md5 dovecot plaintext spa
Routers: accept dnslookup ipliteral manualroute queryprogram redirect
Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
Malware: f-protd f-prot6d drweb fsecure sophie clamd avast sock cmdline
Fixed never_users: 0
Configure owner: 0:0
Size of off_t: 8
Compiler: GCC [4.8.5 20150623 (Red Hat 4.8.5-44)]
Library version: Glibc: Compile: 2.17
                        Runtime: 2.17
Library version: BDB: Compile: Berkeley DB 5.3.21: (May 11, 2012)
                      Runtime: Berkeley DB 5.3.21: (May 11, 2012)
Library version: OpenSSL: Compile: OpenSSL 1.0.2k-fips  26 Jan 2017
                          Runtime: OpenSSL 1.0.2k-fips  26 Jan 2017
                                 : built on: reproducible build, date unspecified
Library version: spf2: Compile: 1.2.11
                       Runtime: 1.2.11
Library version: PCRE2: Compile: 10.23
                        Runtime: 10.23 2017-02-14
Total 11 lookups
WHITELIST_D_MACROS unset
TRUSTED_CONFIG_LIST unset
changed uid/gid: forcing real = effective
  uid=0 gid=0 pid=483
  auxiliary group list: <none>
seeking password data for user "mail": cache not available
getpwnam() succeeded uid=8 gid=12
seeking password data for user "majordomo": cache not available
getpwnam() succeeded uid=992 gid=2
seeking password data for user "diradmin": cache not available
getpwnam() succeeded uid=994 gid=994
seeking password data for user "root": cache not available
getpwnam() succeeded uid=0 gid=0
openssl option, adding to     03104000: 01000000 (no_sslv2 +no_sslv3 +no_tlsv1 +no_tlsv1_1 +cipher_server_preference)
openssl option, adding to     03104000: 02000000 (no_sslv3 +no_tlsv1 +no_tlsv1_1 +cipher_server_preference)
openssl option, adding to     03104000: 04000000 (no_tlsv1 +no_tlsv1_1 +cipher_server_preference)
openssl option, adding to     07104000: 10000000 (no_tlsv1_1 +cipher_server_preference)
openssl option, adding to     17104000: 00400000 (cipher_server_preference)
configuration file is /etc/exim.conf
log selectors = 000024ac e640a066 0000001c
trusted user
admin user
changed uid/gid: privilege not needed
  uid=8 gid=12 pid=483
  auxiliary group list: 12 989 1003
seeking password data for user "majordomo": cache not available
getpwnam() succeeded uid=992 gid=2
seeking password data for user "majordomo": using cached result
getpwnam() succeeded uid=992 gid=2
seeking password data for user "mail": cache not available
getpwnam() succeeded uid=8 gid=12
seeking password data for user "majordomo": cache not available
getpwnam() succeeded uid=992 gid=2
seeking password data for user "mail": cache not available
getpwnam() succeeded uid=8 gid=12
seeking password data for user "mail": using cached result
getpwnam() succeeded uid=8 gid=12
seeking password data for user "mail": using cached result
getpwnam() succeeded uid=8 gid=12
originator: uid=0 gid=0 login=root name=root
sender address = [email protected]
  search_open: iplsearch "/etc/virtual/helo_data"
  search_find: file="/etc/virtual/helo_data"
    key="" partial=-1 affix=NULL starflags=0 opts=NULL
  LRU list:
    5/etc/virtual/helo_data
    End
  internal_search_find: file="/etc/virtual/helo_data"
    type=iplsearch key="" opts=NULL
sender_fullhost = [127.0.0.1]
sender_rcvhost = [127.0.0.1]

**** SMTP testing session as if from host 127.0.0.1
**** but without any ident (RFC 1413) callback.
**** This is not for real!

host in hosts_connection_nolog? no (option unset)
LOG: smtp_connection MAIN
  SMTP connection from [127.0.0.1]
host in host_lookup?
 list element: *
 host in host_lookup? yes (matched "*")
looking up host name for 127.0.0.1
DNS lookup of 1.0.0.127.in-addr.arpa (PTR) succeeded
Reverse DNS security status: unverified
IP address lookup yielded "localhost"
DNS lookup of localhost (A) succeeded
local host found for non-MX address
localhost 127.0.0.1 mx=-1 sort=-64
checking addresses for localhost
Forward DNS security status: unverified
  127.0.0.1 OK
sender_fullhost = localhost [127.0.0.1]
sender_rcvhost = localhost ([127.0.0.1])
set_process_info:   483 handling incoming connection from localhost [127.0.0.1]
host in host_reject_connection? no (option unset)
host in sender_unqualified_hosts? no (option unset)
host in recipient_unqualified_hosts? no (option unset)
host in helo_verify_hosts? no (option unset)
host in helo_try_verify_hosts? no (option unset)
host in helo_accept_junk_hosts? no (option unset)
using ACL "acl_connect"
processing "warn" (/etc/exim.conf 200)
check set acl_c_spam_assassin_has_run = 0
warn: condition test succeeded in ACL "acl_connect"
processing "warn" (/etc/exim.conf 201)
check set acl_m_is_whitelisted = 0
warn: condition test succeeded in ACL "acl_connect"
processing "warn" (/etc/exim.conf 202)
check set acl_c_accept_recipient_if_whitelisted = 1
warn: condition test succeeded in ACL "acl_connect"
processing "warn" (/etc/exim.easy_spam_fighter/connect.conf 1)
check set acl_m_easy69 = 0
check set acl_c_esf_skip = 0
check set acl_m_esf_spf_softfail = 0
check set acl_c_rspamd = 0
warn: condition test succeeded in ACL "acl_connect"
processing "accept" (/etc/exim.conf 204)
check hosts = *
host in "*"?
 list element: *
 host in "*"? yes (matched "*")
accept: condition test succeeded in ACL "acl_connect"
end of ACL "acl_connect": ACCEPT
host in pipelining_connect_advertise_hosts?
 list element: *
 host in pipelining_connect_advertise_hosts? yes (matched "*")
SMTP>> 220 ESMTP Exim
220 ESMTP Exim
smtp_setup_msg entered

A telnet test from the server is below, i'm not sure if there should be more to this? :

Code:
[root@server01 ~]# telnet 74.125.200.26 25
Trying 74.125.200.26...
Connected to 74.125.200.26.
Escape character is '^]'.

Anyone have any ideas please?
 
A telnet test from the server is below, i'm not sure if there should be more to this? :
So from who is that 74.125.200.26 ip address?

Can you try the same using telnet mail.directadmin.com 25? Just to be sure and to see resolving is working too?

Do you only have this issue when mail is send to gmail? Or to every domain?
Did you check that 127.0.0.1 is not has been set in the firewall? There is an option in DA to exclude 127.0.0.1 from the blacklist, but if that is not set, for some odd reasons it can happey, which could maybe explain odd issues.
So check your /usr/local/directadmin/data/admin/ip_blacklist for entry's for your server ip and 127.0.0.1.

You can disable CSF/LFD shortly to see if that helps.
Use the csf -x command as root.
Then use the iptables -L command do check all policy's (should be 3) are set to ACCEPT.
Retry and see if the mail is now sending.
 
So from who is that 74.125.200.26 ip address?

Can you try the same using telnet mail.directadmin.com 25? Just to be sure and to see resolving is working too?

Do you only have this issue when mail is send to gmail? Or to every domain?
Did you check that 127.0.0.1 is not has been set in the firewall? There is an option in DA to exclude 127.0.0.1 from the blacklist, but if that is not set, for some odd reasons it can happey, which could maybe explain odd issues.
So check your /usr/local/directadmin/data/admin/ip_blacklist for entry's for your server ip and 127.0.0.1.

You can disable CSF/LFD shortly to see if that helps.
Use the csf -x command as root.
Then use the iptables -L command do check all policy's (should be 3) are set to ACCEPT.
Retry and see if the mail is now sending.
Thanks for replying, I've no idea what that IP is tbh, it's one that Directadmin Support told me to telnet test port 25.

Result of "telnet mail.directadmin.com 25" is:

Code:
[root@server01 ~]# telnet mail.directadmin.com 25
Trying 167.172.23.237...
Connected to mail.directadmin.com.
Escape character is '^]'.

The issue is sending to any domain.

/usr/local/directadmin/data/admin/ip_blacklist only has 5 entries and does not contain 127.0.0.1 or any server IPs

I've disabled CSF and checked iptables -L which shows:

Code:
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Sending another test after the above with CSF still disabled and it still fails, goes straight into the message queue.
 
Sending another test after the above with CSF still disabled and it still fails, goes straight into the message queue.
Do you use anything else which could interfere with the connection like immunify360 or something?

Because at least at this point we can conclude that port 25 outgoing is open both at server and datacenter level and the timeouts are not caused by the firewall either.

Did you recently change something? Ip address, ipv6, kernel update, something else, before this issue began?

Because it looks like Exim does not want to make connection at the correct place.

Recently we encountered a similar issue, where Exim was trying to connect via the wrong interface which still contained an old ip. Maybe something like that is happening.

Check this thread, there are also various suggestions to test things, maybe by going over them, you can find your cause.
 
Back
Top