How do spammers do this??

[Richard G]

I was checking my Exim mainlog for something and then suddenly I see a lot of lines coming in like this:

2023-09-02 04:26:30 SPFCheck: Soft Fail 95.xx.xx.xx
2023-09-02 04:26:32 H=server.companyhostname.nl [95.xx.xx.xx] incomplete transaction (QUIT) from <[email protected]> for [email protected]

There was a real load of this.
Now the odd thing. This H=server.companyhostname.nl is one of our other servers. But I didn't get any notices of spam from there.

So I quickly thought I would block the server ip and then the mails stopped. And then I checked the exim queue on the other server, but that was empty. So my good guess it seems they manage to use some php script to mail as my other server's hostname is sending the helo.

Or would this be done in another way?

P.s. Since it's night, I checked the php-mail.log files >0kb and there were only 4. Which only 2 or 3 messages, so looks as if it's not php scripts.
And lsof -i:25 does not show any other mailserver running either.

 

[mxroute]

Is the 95.* IP your server or just the hostname? Because if it's not your IP then how they do this is very easy:

"HELO server.companyhostname.nl"

 

[Richard G]

Is the 95.* IP your server or just the hostname?

That's the problem. This 95.* ip belongs to H=server.companyhostname.nl which is from our other server, so they are not spoofing it.
The 95* ip is that server's primary ip, so also the hostname ip of that server.

 

[sparek]

Someone or some process from the 95.xx.xx.xx server is connecting to this server and doing a partial SMTP transaction but ending it with a quit.

EHLO something
mail from: <[email protected]>
rcpt to: <[email protected]>
quit

You can test this by doing an SMTP transaction through a basic telnet session from the 95.xx.xx.xx server

telnet %whateverthisserversipaddressis% 25

and then in the telnet session:

EHLO something
mail from: <[email protected]>
rcpt to: <[email protected]>
quit

Do you have sender callouts enabled on the 95.xx.xx.xx server? This could be the 95.xx.xx.xx calling back to %whateverthisserversipaddressis% to make sure that the person sending the email - [email protected] - actually exists.

There's nothing inherently malicious with any of this. Unless you consider checking for the existence of the [email protected] email address to be malicious.

 

[mxroute]

That's the problem. This 95.* ip belongs to H=server.companyhostname.nl which is from our other server, so they are not spoofing it.
The 95* ip is that server's primary ip, so also the hostname ip of that server.

Is that server configured to relay mail through the server you caught the log on? It could just be a logging issue that you don't see it on the 95.* server. Exim does a strange thing when configured to relay mail through another server, it'll actually open a connection with the relay and declare the sender/recipient even when it rejects the email. I'll prove my statement, using the exact same botnet that you identified in this event (I've been tracking it for a while). Check this out:

1. Exim on eagle.mxlogin.com is configured to relay all mail through filtergroup.mxroute.com

2. Log on eagle.mxlogin.com:

2023-08-23 15:07:04 H=110-175-220-250.static.tpgi.com.au (203-194-21-194.tpgi.com.au) [110.175.220.250] F=<[email protected]> rejected RCPT <mlnomeq64hff@{censored}>: authentication required

3. Log on filtergroup.mxroute.com:

Aug 23 15:07:04 filter006 postfix/smtpd[4911]: NOQUEUE: reject: RCPT from unknown[45.43.208.25]: 450 4.1.2 <mlnomeq64hff@{censored}>: Recipient address rejected: Domain not found; from=<[email protected]> to=<mlnomeq64hff@{censored}> proto=ESMTP helo=<eagle.mxlogin.com>

Now, why exim feels the need to open a connection to it's outbound relay and declare it's sender/recipient when it rejected the email itself, I do not quite understand. It's gotta be something about order of operations, but I just don't get it. So I pretty much ignore it. But since I don't understand it, there may be variations of this event that trigger similar results without matching the configuration I use.

That said, there's another fun thing here. What you've seen is a botnet that has persisted for months. Every IP connects and attempts exactly 96 recipients on a domain, then is never seen again. You can see my list of IPs I've caught in this botnet here: https://github.com/mxroute/da_server_updates/blob/master/sec/botnet.list

That IP list gets new additions at least every few hours, every single day, for months now.

 

[Richard G]

Someone or some process from the 95.xx.xx.xx server is connecting to this server and doing a partial SMTP transaction but ending it with a quit.

That's what I thought too...... someone or some process on the 95.* server must be trying this. Problem is, I can't find any proof of it in the 95.* server, not in exim logs and neither in php-mail.log files of users. And that is the problem I'm having with this, that I can't find where it's coming from.
Even Maldetect has not found anything yet.
Sender callouts... if that is not enabled by default then no I don't have it on any server.

There's nothing inherently malicious with any of this. Unless you consider checking for the existence of the [email protected] email address to be malicious.

Yes it is malicious because it shouldn't come from a non existing mail address on the 95.* server anyway. Next to that, it's not 1 address of customerdomain.nl but things like [email protected] [email protected] and so on, like 50 of them at 1 time. To me that is malicous.
It's normal that spammer try this, but then they directly login or try to send to the vps were customerdomain.nl resides and then I wouldn't be worried.
But in this case, my own 95.* servers is trying to send mail from non-local addresses to my external vps on the 173.* ip. That is relaying and should not be possible. Or it's something from my own 95.* server, and then it's malicious too.

Is that server configured to relay mail through the server you caught the log on?

Not that I know of. It's a default exim.conf. No relaying customisations.
The server I cought the log on is a VPS we are temporarily using with a 173 ip address.

That botnet list is interesting, you can add 46.148.40.* to that list.

2023-08-27 00:45:58 login authenticator failed for (localhost) [46.148.40.195]: 535 Incorrect authentication data (set_id=netops)
2023-08-27 00:50:46 login authenticator failed for ([185.36.81.16]) [185.36.81.16]: 535 Incorrect authentication data (set_id=bank)
2023-08-27 00:51:05 login authenticator failed for (localhost) [46.148.40.195]: 535 Incorrect authentication data (set_id=metabase)
2023-08-27 00:56:19 login authenticator failed for (localhost) [46.148.40.195]: 535 Incorrect authentication data (set_id=katalog)
2023-08-27 01:00:08 login authenticator failed for (User) [79.110.62.182]: 535 Incorrect authentication data (set_id=user)
2023-08-27 01:01:09 login authenticator failed for (localhost) [46.148.40.195]: 535 Incorrect authentication data (set_id=manage-vps)
2023-08-27 01:06:15 login authenticator failed for (localhost) [46.148.40.195]: 535 Incorrect authentication data (set_id=mvp)
2023-08-27 01:06:35 login authenticator failed for (localhost) [46.148.40.77]: 535 Incorrect authentication data (set_id=5555)
2023-08-27 01:10:54 login authenticator failed for (localhost) [46.148.40.195]: 535 Incorrect authentication data (set_id=myriam)
2023-08-27 01:11:01 login authenticator failed for (localhost) [46.148.40.94]: 535 Incorrect authentication data (set_id=ky)
2023-08-27 01:12:01 login authenticator failed for (localhost) [46.148.40.77]: 535 Incorrect authentication data (set_id=esf)
2023-08-27 01:16:09 login authenticator failed for (localhost) [46.148.40.94]: 535 Incorrect authentication data (set_id=rekrutacja)
2023-08-27 01:16:22 login authenticator failed for (localhost) [46.148.40.195]: 535 Incorrect authentication data (set_id=page_options)
2023-08-27 01:16:58 login authenticator failed for (localhost) [46.148.40.77]: 535 Incorrect authentication data (set_id=columbia)
2023-08-27 01:21:07 login authenticator failed for (localhost) [46.148.40.94]: 535 Incorrect authentication data (set_id=pos)
2023-08-27 01:21:29 login authenticator failed for (localhost) [46.148.40.195]: 535 Incorrect authentication data (set_id=miffy)
2023-08-27 01:21:56 login authenticator failed for (localhost) [46.148.40.77]: 535 Incorrect authentication data (set_id=pannello)
2023-08-27 01:26:16 login authenticator failed for (localhost) [46.148.40.94]: 535 Incorrect authentication data (set_id=Aries)
2023-08-27 01:26:26 login authenticator failed for (localhost) [46.148.40.195]: 535 Incorrect authentication data (set_id=nizhnevartovsk)

etc..
Got a couple of hundreds of them in the log. But these are "normal" attacks.

As for my issue, I did notice that they were trying the same customerdomain.nl every time. Also with .ru mail address like this:

2023-09-02 19:17:47 H=server.mycompanyhostname.nl [95.xx.xx.xx] incomplete transaction (QUIT) from <[email protected]> for [email protected]

So same as the .cz but now with .ru from address.

 

[Richard G]

Ah... I found some proof on the 95.* server in the Exim rejectlog.
I checked the 19:17:47 time and found 3 lines:

uired
2023-09-02 19:17:47 H=([36.105.172.100]) [36.105.172.100] F=<[email protected]> rejected RCPT <[email protected]>: authentication required
2023-09-02 19:17:47 H=([36.105.172.100]) [36.105.172.100] F=<[email protected]> rejected RCPT <[email protected]>: authentication required
2023-09-02 19:17:47 H=([36.105.172.100]) [36.105.172.100] F=<[email protected]> rejected RCPT <[email protected]>: authentication required

So they are using the Exim server on the 95.* server, however, on that server the only thing residing of customerdomain.nl are some old certificates and 4 log files, nothing else.

And now the interesting part... from the Exim mainlog on the 95* server, same time exactly, 19:17:47.

2023-09-02 19:17:47 BlacklistCheck: Blacklisted address - 36.105.172.100
2023-09-02 19:17:47 H=([36.105.172.100]) [36.105.172.100] F=<[email protected]> rejected RCPT <[email protected]>: authentication required
2023-09-02 19:17:47 [173.xx.xx.xx] SSL verify error (during R-verify for [36.105.172.100]): certificate name mismatch: DN="/CN=vps.receivingvps.nl" H="mail.customerdomain.nl"
2023-09-02 19:17:47 BlacklistCheck: Blacklisted address - 36.105.172.100
2023-09-02 19:17:47 H=([36.105.172.100]) [36.105.172.100] F=<[email protected]> rejected RCPT <[email protected]>: authentication required
2023-09-02 19:17:47 [173.xx.xx.xx] SSL verify error (during R-verify for [36.105.172.100]): certificate name mismatch: DN="/CN=vps.receivingvps.nl" H
="mail.customerdomain.nl"
2023-09-02 19:17:47 BlacklistCheck: Blacklisted address - 36.105.172.100
2023-09-02 19:17:47 H=([36.105.172.100]) [36.105.172.100] F=<[email protected]> rejected RCPT <[email protected]>: authentication required

So it's coming from external, and using my 95* server to send mail to my 173.x VPS... I'm confused as to how this is possible. And also how come a blacklisted address still can try to send mail.
The "vps.receivingvps.nl" with the 173 ip is where I found all those logs with attempts coming from my 95* server.

 

[sparek]

Are you allowing under-privileged users on the 95.xx.xx.xx server to make outbound port 25 connections? This would make 95.xx.xx.xx susceptible to Dark Mailer style exploits.

Regardless... it doesn't really stop someone from another server that you don't control from constantly trying to find valid email addresses on your server. And from a structural point of SMTP, there's nothing meant to stop such attempts - it's basically how email works.

If the malicious user from a remote server uses the constant barrage of finding a valid email address and then actually sends an email to that address... what's your server suppose to do? Say no? There's nothing inherent in SMTP that would stop this.

 

[Richard G]

Are you allowing under-privileged users on the 95.xx.xx.xx server to make outbound port 25 connections?

Not that I know of. As said it's a default exim setup.

Regardless... it doesn't really stop someone from another server that you don't control from constantly trying to find valid email addresses on your server.

I think you miss something, because that is not what's happening. Well... also, but that is not the issue I'm worried about.
They are trying via server A, to reach mail addresses on server B, which should never be possible unless the server is allowing relaying, which is not the case.

what's your server suppose to do? Say no?

That is exactly what should be the case.
Domain is not present on server A. So when they are sending to a non-exising domain on server A, then server A should say "does not exist, goodbye" and not "oh wait. I'm going to send to the correct server (server B) where the domain is residing", right?
And that is what's happening somehow according to the logs.

 

[Driesp]

These connections must come from the server with 95.xx.xx.xx.
It is 'impossible' to spoof this IP and using a spoofed IP to make tcp connections.

Maybe a PHP script that is allowed to create tcp sockets towards external IP's?

 

[Richard G]

It is 'impossible' to spoof this IP and using a spoofed IP to make tcp connections.

No, it's not spoofed indeed. Check the main and reject logs I placed. They are logging in remotely to 95.xx with mail addresses non existing on the 95 server and then the 95xx is trying to send the mail to the 173 server.
Mainlog and rejectlog of the 95 server are in post #7.

The log of the receiving 173 server is the last code line in post #6.

If it was a php script, I probably woulnd't see this in the Exim logs, or I should encounter it in any of the php-mail.log files of the users, right?

 

[Driesp]

I should encounter it in any of the php-mail.log files of the users, right?

No, not necessarily.
php-mail.log only logs php mail() e-mail.
If php is built with tcp sockets, a PHP script can set up a tcp socket. And using smtp commands sent e-mail.
It could also be a shell, perl or python script if apache allows cgi scripts.

Personally I have disabled sockets and posix in the PHP build config (--disable-posix --disable-sockets)
And disabled cgi (--disable-cgi) in the apache build config.

I have enabled SMTP_BLOCK via csf.
This would only allow the root user to connect towards 25,465,587.

But they need a working user and password to be able to sent e-mail via your other server, right?

 

[Richard G]

Servers is not relaying. I just tried from another server from us like this:

mail from: [email protected]
250 OK
rcpt to: [email protected]
550-Verification failed for <[email protected]>
550-Unrouteable address
550 Sender verify failed

This is more like expected. And this doesn't even reach the 173 server. So I keep wondering....

I also have disabled cgi on the server, and perl access is very limited.
I don't see anything strange with ps faux either.

I also have SMTP_BLOCK = "1"
but also:
SMTP_ALLOWLOCAL = "1"
so sites can use php mail.

But i don't think it's some script, it's Exim... Look:

2023-09-02 19:17:47 H=([36.105.172.100]) [36.105.172.100] F=<[email protected]> rejected RCPT <[email protected]>: authentication required

this is the Exim mainlog on the 95 server. So they are logging in and they get the correct error. But then....

Exact same time on the 173 vps in Exim's mainlog:

2023-09-02 19:17:47 H=server.mycompanyhostname.nl [95.xx.xx.xx] incomplete transaction (QUIT) from <[email protected]> for [email protected]

as you can see, exactly the same time, sender address and receiving address.

So the question is, when i do this from an external server, I also get either like above the verification failed and unroutable address or when done in another way I get the "authentication required", but nothing is seen in the mainlog of the 173 vps.

When they do the same, they are blocked on the 95* but still it appears as attempt in the 173 log. That is what I don't get.

 

[sparek]

I think you miss something, because that is not what's happening. Well... also, but that is not the issue I'm worried about.
They are trying via server A, to reach mail addresses on server B, which should never be possible unless the server is allowing relaying, which is not the case.

Yea, I'm definitely missing something.

Are you saying that Server A should never, ever, ever send emails to Server B (95.xx.xx.xx)?

You need to either block 95.xx.xx.xx from %whateverthisserversipaddressis%

You need to either block outbound connections on port 25 on the %whateverthisserversipaddressis% to 95.xx.xx.xx

Or you need to block inbound connections from %whateverthisserversipaddressis% on port 25 on the 95.xx.xx.xx server.

I'm missing something because I don't really know what you're concerned about.

 

[sparek]

I would check to see if under privileged users on the %whateverthisserversipaddressis% are able to make outbound connections on port 25. If they are then %whateverthisserversipaddressis% would be susceptible to abuse.

SSH into %whateverthisserversipaddressis% and switch to any under-privileged user on the server:

su -s /bin/bash - %underprivilegeduser%

Then from that prompt, type:

telnet 95.xx.xx.xx 25

If it connects, then you are susceptible. If it doesn't then you're not.

A Dark Mailer script used this technique many years ago. It allowed under-privileged users to send out spam from the server without any logs in the mail server, effectively because the Exim server was being bypassed entirely.

 

[mxroute]

So it's coming from external, and using my 95* server to send mail to my 173.x VPS... I'm confused as to how this is possible. And also how come a blacklisted address still can try to send mail.

It's definitely the same botnet and same activity that I've been tracking for a few months, so I'm at least a little bit ahead of you on this but I still don't have the whole thing wrapped up in a nice package with a full conclusion. Since you're not relaying one server through the other, I'm going to suggest something slightly different.

Let's have the origin server, your 95.* server noted in the original log in this post be called Server1.

Let's have the server that you first noticed the exim logs on noted as Server2.

Now, they tried to send an email (actually, probably 96 at a time, if you count them) to an invalid address on Server1. Server1 opened a connection with Server2 and passed on the sender/recipient information despite the fact that Server1 rejected the email. This is why it's weird, let's continue.

If that email were to have been successfully delivered to Server1, would Server1 have suddenly had any obligation whatsoever to open a connection with Server2 and if so, why? That Server2 is yours and not just some random person's server, that Server1 is opening a connection with Server2 must mean that there is some connection between the two, at least in reference to that recipient domain. Find that connection, and you'll at least be up to speed on this issue and maybe we'll both learn something interesting about it.

Because for me it was because my Server1 relays all outbound mail through my Server2, so Server1 had at least some reason to open a connection with my Server2. I feel like if we understand why your Server1 reached out to your Server2, and it isn't the same reason mine had to do the same, we're going to discover an exploitable vulnerability here. Be it of exim or of our configs. Since we both own our Server2s in these cases, it's not that bad. But what if the reason for your Server1 opening the connection with your Server2 is actually an external factor, which means that our servers could technically be opening similar connections with other people's mail servers and because exim doesn't even log that it does so, we have no knowledge of it happening?

 

[Richard G]

Are you saying that Server A should never, ever, ever send emails to Server B (95.xx.xx.xx)?

No, certainly not.
I'm saying that Server A (95.x.x.x) should never ever send mails to server B (173.x.x.x) from unknown senders or non-local domains.
Domains on server A should ofcourse be able to mail to server B, so a block is no solution at all. And this is normal email operation.

It's some Russian domains (and czech ones) who are connecting to server A and I'm getting logs in server B. I presume you have seen post #13 with the 2 loglines right? It makes it very clear what is happening.

I know about a dark mailer script, but....

effectively because the Exim server was being bypassed entirely.

Which is not happening otherwise I wouldn't see it in the Exim logs, they would not connect via Exim.
Have a look at the first log in post #13 which is from server A and then the second logline which is from server B, that says it all.
It's all Exim, so in this case Exim is not bypassed at all, otherwise I wouldn't have Exim logs.

su -s /bin/bash - %underprivilegeduser%

Then from that prompt, type:

telnet 95.xx.xx.xx 25

I presume an underprivilegeduser is just one of the customers accountnames?

/home]# su -s /bin/bash - username
Last login: Sat Sep  2 00:13:53 CEST 2023
username@server:~$ telnet mail.server-on-173.nl 25
Trying 173.xx.xx.xx...
telnet: connect to address 173.xx.xx.xx: Connection refused
Trying 2a02:xxxx:xxxx:xxxx::1...
telnet: connect to address 2a02:xxxx:xxxx:xxxx::1: Network is unreachable

which was to be expected, because the Exim log state that external users (czech, russion etc.) are connecting via Exim to the 95.* server.

 

[Richard G]

Let's have the origin server, your 95.* server noted in the original log in this post be called Server1.

Yes, or server A and the 173 (vps) will be server 2 or server B. I totally agree with you that it's the same botnets, as several of the ip's listed in both Exim logs were also present in your botnet list.

This is why it's weird, let's continue.

Exactly, that is wat the 2 loglines in post #13 are showing and what is amazing me.
As far as I know I don't allow relaying and if I try myself from another server, I'm not allowed either.

Find that connection, and you'll at least be up to speed on this issue and maybe we'll both learn something interesting about it.

I'm eager to learn here too because it's interesting to investigate how to block suspicous behaviour.
As for now, I've seen a lot of this botnet, the odd thing is that only 1 customerdomain is mailed to.
I doublechecked on the old server (server 1 or A) where the domain has resided before. But nothing is to be found anymore from that domain, not in /var/named, not in /etc/virtual/ neither in the domains or domainowners file. Only some certificate logs and other logs, nothing more.

However, it's interesting as to why they only attack this 1 domain. It's a user with a small phpbb forum.

which means that our servers could technically be opening similar connections with other people's mail servers and because exim doesn't even log that it does so, we have no knowledge of it happening?

Exactly. I'm afraid of some unkown bug they managed to discover.
If you want (and are interested) I'm prepared to give you all the things you need. Like ip's so you can do some telnet tests if you want.

But as said, I just have a default exim.conf file, with the only adjustments in a custom file that I use different RBL's and bounce_return_message = false but nothing which could interfere with mail delivery in a normal way.

 

[sparek]

It's some Russian domains (and czech ones) who are connecting to server A and I'm getting logs in server B. I presume you have seen post #13 with the 2 loglines right? It makes it very clear what is happening.

OK, I think I see what you're saying now.

My initial thought would be that you have a verify = recipient/callout some where in Server1 - the one the Russians are connecting to. Some where in one of the Exim configuration files, that shouldn't be there.

It would seem to me like Server1 is doing a callout to verify that the recipient exists before concluding that no SMTP Authentication information was given. I would think that this is a configuration error.

Server1 is getting the connection from the Russians. They are not doing an AUTH. Getting to RCPT TO at which point Server1 does a callout request to Server2 to see if that recipient actually exists. Then it's failing because there is no AUTH.

 

[Ohm J]

At this point, If you wanna prevent botnet, you should config at Firewall like... csf firewall blocklist option. ( you can put your own blocklist URL ).

if just config at Exim, they still create the connection without care success or not. But just make your server hight load.