4Gigs of email bandwidth in 5 days

WholesaleDialup

WholesaleDialup

Verified User
Joined
Sep 25, 2004
Messages
178
Location
San Antonio, TX
OK, so I have this user who has one domain on our DA server and 4 or 5 email accounts.

I allow them 5gb of bandwidth per month, then the site gets suspended.

I have 50 or so domains on this server in they are all limited to 5gb per month, none ever get over 1gb, ever.

This domain however has consistant issues with getting suspended due to bandwidth limits being reached. This has been going on for a LONG time.

The last time this happened, I was for sure it was due to a compromised PHP script on their site, I made them take it all down and re-work it since they did in fact have insecure php mailer code that I found. I never did nail down exactly 100% for sure what the issue was except that the bandwidth usage was related to email. I made this determination based on the bandwidth graphs provided by DA but the Exim logs did not result in anything concrete.

After they put their new, better, more secure site back up, things were good for a few weeks then the other days SMACK!, they blew over their bandwidth usage almost entirely in a 5 day period. According to the DA graphs, it's all email bandwidth, not apache or anything else.

Here is a link to the graph which I took a screen shot of and uploaded:
http://secure.muchoweb.com/temp/bandwidth-screenshot.jpg

I have spent two days grepping all the Exim logs, googling on different spam detection methods from the logs etc.. No Joy! I have even changed the logging options to include the path info to see if a script is doing the spamming.

I have grepped for "A=" using regular expressions (Wild Cards) to see if I could see a particular user of the domain doing all the sending.

I just can't find anything that makes sense or that would cause this much email bandwidth to be used, the number of email being sent to/from this domain just don't equal up to the amount of bandwidth DA shows in the graph.

Is it possible the graph is wrong?

What else can I do to find this never ending problem?

Pulling my hair out for months on this, any help would be Oh So appreciated.

Thanks in advance!
 
It may still be a PHP script used to send spam, happens very often.
Any RFI (Remote File Include) vulnerability can serve that, not just a faulty PHP mailer.

To find the right lines (messages injected in exim from a local user) in the exim log use `zgrep "U=<username> P=local" /var/log/exim/mainlog*', for example:
Code: 
# zgrep 'U=noirgame P=local' /var/log/exim/mainlog*
/var/log/exim/mainlog:2008-11-20 01:18:48 1L2xFw-0006oB-Oo <= noirgame@xxx U=noirgame P=local S=1133 [email protected] T="Nuovo Ticket di Supporto [#606196]" from <noirgame@xxx> for xxx@xxx
/var/log/exim/mainlog:2008-11-20 01:18:48 1L2xFw-0006oD-Qr <= noirgame@xxx U=noirgame P=local S=1481 [email protected] T="Avviso di nuovo ticket" from <noirgame@xxx> for xxx@xxx
[...]

Sadly this won't help you find the vulnerable script, but often spam scripts have a recipients file saved in the same path, therefore you can `grep -Ri <recipient> /home/username/public_html' and locate at least the directory. If there is a known web software there, check on http://www.milw0rm.org/ if there is some kind of vulnerability and ask the customer to update it. If it's a custom software check for unsafe open() system() exec() fopen() shell() etc etc.
 
Last edited:
Looking at how many emails were actually sent, it doesn't appear to be spam (though I could be wrong). It looks more like they're sending some emails with large attachments. Odd that all the email traffic was only during those days though.
 
tillo said:
It may still be a PHP script used to send spam, happens very often.
Any RFI (Remote File Include) vulnerability can serve that, not just a faulty PHP mailer.

To find the right lines (messages injected in exim from a local user) in the exim log use `zgrep "U=<username> P=local" /var/log/exim/mainlog*', for example:
Code: 
# zgrep 'U=noirgame P=local' /var/log/exim/mainlog*
/var/log/exim/mainlog:2008-11-20 01:18:48 1L2xFw-0006oB-Oo <= noirgame@xxx U=noirgame P=local S=1133 [email protected] T="Nuovo Ticket di Supporto [#606196]" from <noirgame@xxx> for xxx@xxx
/var/log/exim/mainlog:2008-11-20 01:18:48 1L2xFw-0006oD-Qr <= noirgame@xxx U=noirgame P=local S=1481 [email protected] T="Avviso di nuovo ticket" from <noirgame@xxx> for xxx@xxx
[...]

Sadly this won't help you find the vulnerable script, but often spam scripts have a recipients file saved in the same path, therefore you can `grep -Ri <recipient> /home/username/public_html' and locate at least the directory. If there is a known web software there, check on http://www.milw0rm.org/ if there is some kind of vulnerability and ask the customer to update it. If it's a custom software check for unsafe open() system() exec() fopen() shell() etc etc.
Click to expand...


Thanks, I issued this exact command:
zgrep 'U=kasco00417 P=local' /var/log/exim/mainlog*

kasco00417 is the DirectAdmin username which owns the domain where this problem exist.

Was that the username I was suppose to use in the string or was I suppose to use an email address?

When I issued this command there were no results at all.

I did some grep commands earlier searching for things like "/home" etc so I could see anything that was fired from any of the web directories under home but no luck there either. :confused:
 
Dravu said:
Looking at how many emails were actually sent, it doesn't appear to be spam (though I could be wrong). It looks more like they're sending some emails with large attachments. Odd that all the email traffic was only during those days though.
Click to expand...

Well, that's kind of what I thought.. I will dig around a little more, I saw some email subjects with "Video" in them, I will inspect them a little more.
 
Can someone offer a grep example or zgrep that will allow me to search for emails over a certain size?

I think it's S= something.. I know where the size is but how can I grep for only emails with attachments over a certain size etc.. ?

Thanks to all for your help so far.
 
zgrep 'S=[0-9]\{8,\}' /var/log/exim/mainlog

I dont know what format the size really is in. If its bits or bytes or what. This should grab anything with 8 characters or more after the S=.

You might want to change that 8 to a 7 then it would grab anything 1mb or bigger. But I am just guessing here :D
 
It would be:

Code: 
zgrep 'remote_smtp' /var/log/exim/mainlog* |sed 's/=/ /g' |awk '{if ((($13 + 0) == $13) && ($13 > 10000)) print; else if ((($14 + 0) == $14) && ($14 > 10000)) print}'

Just change both occurrences of 10000 with the threshold.

I had to hack it a bit to make it work with the senseless exim log... :( be sure to copy/paste it correctly.
 
tillo said:
I had to hack it a bit to make it work with the senseless exim log... :( be sure to copy/paste it correctly.
Click to expand...
tillo, I'm the guy who decided what and how exim should log on DirectAdmin servers; it fit my needs at the time :).

What would you suggest be changed?

Specifically check your exim.conf file for what I log, and I'll be happy to take your suggestions.

Thanks.

Jeff
 
Hello,

The actually bandwidth counting that DA itself does is done via the /etc/exim.pl script, and logs everything in:
/etc/virtual/usage/username.bytes

So all emails sent by the account (anything under the account, apache, him, pop, etc), as well as any emails that arrive into inboxes.. will all be logged in there.
You can see the message ID for each message in that file.
If you don't see much detail in the bytes file, then update your exim.pl file:
wget -O /etc/exim.pl http://files.directadmin.com/services/exim.pl

then restart exim.

Once a day, during the nightly tally, DA will dump the bytes file into:
/usr/local/directadmin/data/users/username/bandwidth.tally

Then after everyting is in the bandwidth.tally file (for all days of the month) DA goes through it and counts it all up giving you the big total.

One issue that can happen if you had an old exim.pl file, is that if any email is stuck in you outbound exim mail queue and can't get out, then each attempted send gets counted in the bytes. The reason we have a newer exim.pl is that it logs the message ID which is unique, thus DA can identify any repeats, and only count it once.

So basically, just make sure you have the newest exim.pl.

If you had the old one, then all you can do is compare bytes sizes from the bytes log and the exim mainlog... but if you have the new exim.pl, then you can match up any message ID in the logs to confirm that they all match up with real emails.

John
 
Don't worry Jeff, it's not your fault and I don't think anyone could make exim create a "better" log just by editing exim.conf.

It's just that "very useful" logs should be CSV or CSV-like, like Apache's logs.
I should be able to use `awk' on them and be sure (after setting the separator and string boundaries) that at position X there will be a certain value. Exim puts "address" or "name <address>" for the sender or recipient address, and spaces are already used to separate every field... they should use commas or tabs instead, otherwise set string boundaries (like " or ') for any address like they do with the subject.

I guess that's the reason for which there is an exim.pl helper but not an apache.pl helper in DA :D forgot about that, John is right.

The perfect command would be:
Code: 
sed 's/[=&]/ /g' /usr/local/directadmin/data/users/<username>/bandwidth.tally |awk '{if (($1 > 10000) && ($3 == "email")) print}'
This will show address and ID of any incoming or outgoing 10KB+ email for a certain user. You can then 'zgrep' the ID from the mainlog file if anything interesting.
 
Last edited:
tillo said:
Don't worry Jeff, it's not your fault and I don't think anyone could make exim create a "better" log just by editing exim.conf.
Click to expand...
I'm not worried :) and I'm not taking it personally.

Just asking for help anywhere I can get it :D.

I first wrote the SpamBlocker exim.conf file for myself. Then put it out on my website in case anyone else wanted to use it. In those days the exim.conf file included with DirectAdmin was still accepting all email and bouncing undeliverables later, so I had to do something for myself.

And don't blame JBMC staff for that; they were using the standard exim.conf file delivered at the time.

Mine was obviously a bit better and John dedided to make it a part of DirectAdmin. So what began as a labor of necessity became what could be a full-time operation if I had the time.

Jeff
 
Wow! Thanks for all the great responses..

Hopefully I will have time tonight or tomorrow to dig in and try some of these out.

Not sure if the logs that showed the trouble have fallen off and been rotated out yet or not but I will soon find out. I think I dowloaded them to my local machine anyhow.

Again, thanks for the responses, I am sure they will help others as well.:D
 
Sweet, using the info DA Support provided above I was able to find the issue.

I basically just looked at the bytes file for today to see if the message ID existed first, it didn't. So, I updated the exim.pl as instructed to make life easier with these types of issues in the future. Also, to make sure the bandwidth is being calculated correctly.

Since I didn't have a message ID to work with, I just started looking through the tally file for the user, looking for large emails sent. I found a ton from two different email addresses with almost the same file size.

For example:
5694854=type=email&[email protected]&method=outgoing
5693503=type=email&[email protected]&method=outgoing
5694854=type=email&[email protected]&method=outgoing
5693503=type=email&[email protected]&method=outgoing
5694854=type=email&[email protected]&method=outgoing
5693503=type=email&[email protected]&method=outgoing

So, I then did something like this:
zgrep 'S=5693503' /var/log/exim/mainlog*

with each of the email sizes and noticed that all of these seemed to be emails with the same subject and they were all outgoing emails from these same two users.

It looks like it was probably some stupid joke email that got clogged in the queue and got counted more than it should have been, also there was some legit bandwdth usage there as well.

Since there is at least a chance that the bandwidth shown was not correct, I just fixed the issue by downloading the new exim.pl and gave them some more bandwidth to get them through the month.

If it happens again, I know right where to look. I know have better logging in place because I changed some of the options, better exim.pl and so on. Plus, I know a LOT more about how to hunt down this sort of issue within DA/Exim.

Thanks a ton for all your help and I hope you all have a great week and a good Thanksgiving to all those that celebrate it. :D:p:)
 
You must log in or register to reply here.
Back
Top