number of outgoing emails is incorrect

G

gate2vn

Verified User
Joined
Nov 9, 2004
Messages
521
Location
Oslo
A customer gets a warning that their account has reached 20K email / day sending out. However, when checking in /etc/virtual/usage/username.bytes, the total outgoing emails in the last 25 hours are around 7934 emails, not 20000.

I am using this command for checking the going emails
grep "method=outgoing" /etc/virtual/usage/username.bytes | grep -oP 'email=\K[^&]+' | sort | uniq -c | sort -nr
Click to expand...

How to debug to find out the reason?
Thanks.
 
Usage at the user level is giving out warning. That's why I double check in /etc/virtual/usage. But then the numbers are different. That's why I want to find out the reason.
 
gate2vn said:
A customer gets a warning that their account has reached 20K email / day sending out. However, when checking in /etc/virtual/usage/username.bytes, the total outgoing emails in the last 25 hours are around 7934 emails, not 20000.

I am using this command for checking the going emails


How to debug to find out the reason?
Thanks.
Click to expand...

Maybe....starting from update 1.702, sent email limits also count local-to-local messages.
Previously, emails sent between local accounts on the same server were not counted toward the daily sending limit
I'm also looking for a way to get rid of this.
 
@gate2vn, the /etc/virtual/usage/username.bytes file contains one for each unique email ID. However single email can have multiple recipients.

Old Exim configuration used to account each recipient individually. New Exim configuration counts only unique message IDs for bandwidth (the /etc/virtual/usage/username.bytes file) but total number of recipients for the send limit (the /etc/virtual/usage/username file).
@DVD we can add customisable condition to optionally disable the local sends accounting.

This would allow you to continue ignoring local-to-local emails. However, in the long term I think it would be better to adjust the limits accordingly and not rely on local sends to be exempt from accounting.
 
We also prefer the old method of accounting for mails, which does not include local delivery.
Custom behavior would be a great option.
 
fln said:
....However, in the long term I think it would be better to adjust the limits accordingly and not rely on local sends to be exempt from accounting.
Click to expand...

Nothing to say ! Flawless !
But in ipotetic scenarios, this behavior causes major issues.

For instance, consider a case where info@ (or accounting@, or ...) receives 100 legitimate emails from internal/external

Since this address acts as an alias/forwarder (es. a certain department has 5 employees who all need to have the same communication personally) that automatically forwards messages to 5 internal accounts (user1@, user2@, etc.)... so a single internal email instantly counts as 5 sends, quickly draining the daily limit...
 
I am not sure I fully understand the scenario you are describing. Things to keep in mind about the current Exim configuration:
The sent email counting happens before Exim routes the email (expands the forwarders).

It is possible to use up the sent email limit if email is sent from [email protected] to [email protected], [email protected], [email protected], ... In other words if instead of writing an email to the forwarder address the sender would instead add multiple recipients in the To: field.

The send limit could be drained in the following scenario:

User [email protected] writes an email:

Code: 
From: [email protected]
To: [email protected], [email protected], [email protected], [email protected]

This first email arriving from external host does not count into any send limits. However when [email protected] hits Reply To All button on his email client, he sends the following email:

Code: 
From: [email protected]
To: [email protected], [email protected], [email protected], [email protected]

Such email would increase the john mailbox send limit by 4, because he sent an email to 4 recipients.


The local send exemption flag I was planning to add would not help in mixed local/external recipients scenario. The flag for disabling local sends would look like this:

Code: 
APPLY_SEND_LIMIT = ${if forany{$recipients_list} { !match_domain{${domain:$item}}{+local_domains} }}

This Exim config blob could be translated as - count sent emails as usual if at least one of the recipients domain is not managed by this server.

This would mean the example above would still bump the mailbox [email protected] by 4 because one of the recipients is external.


Another thing to keep in mind. The main reason for counting local-to-local sends is to make sure if a company A and company B happens to be using the same server, the emails between them would be properly counted.

It seems too me that you are more interested in not counting the emails from same domain to same domain, not emails from same server to same server (different domains). The old Exim behaviour was to not count emails to the same server.
 
  • Like
Reactions: DVD
If [email protected] has e.g. an automated tool that sends a lot to e.g. [email protected], you don't want to be forced to increase his outgoing send limit, unless you can still limit the real outgoing mail to anything but *@domain.com.
This is imho safer regarding spam-runs and hacked mail accounts (1000 mail spike anomalies to internal accounts should be visible, but is less worse than 1000 spammails going external). But counting is still good, even for internal mails. After all it's a service on your server and you want to see numbers.
 
fln said:
I am not sure I fully understand the scenario you are describing. Things to keep in mind about the current Exim configuration:
The sent email counting happens before Exim routes the email (expands the forwarders).

It is possible to use up the sent email limit if email is sent from [email protected] to [email protected], [email protected], [email protected], ... In other words if instead of writing an email to the forwarder address the sender would instead add multiple recipients in the To: field.

The send limit could be drained in the following scenario:

User [email protected] writes an email:

Code: 
From: [email protected]
To: [email protected], [email protected], [email protected], [email protected]

This first email arriving from external host does not count into any send limits. However when [email protected] hits Reply To All button on his email client, he sends the following email:

Code: 
From: [email protected]
To: [email protected], [email protected], [email protected], [email protected]

Such email would increase the john mailbox send limit by 4, because he sent an email to 4 recipients.


The local send exemption flag I was planning to add would not help in mixed local/external recipients scenario. The flag for disabling local sends would look like this:

Code: 
APPLY_SEND_LIMIT = ${if forany{$recipients_list} { !match_domain{${domain:$item}}{+local_domains} }}

This Exim config blob could be translated as - count sent emails as usual if at least one of the recipients domain is not managed by this server.

This would mean the example above would still bump the mailbox [email protected] by 4 because one of the recipients is external.


Another thing to keep in mind. The main reason for counting local-to-local sends is to make sure if a company A and company B happens to be using the same server, the emails between them would be properly counted.

It seems too me that you are more interested in not counting the emails from same domain to same domain, not emails from same server to same server (different domains). The old Exim behaviour was to not count emails to the same server.
Click to expand...

Yes ... yes but it was really the first scenario that came to mind (definitely wrong), let's imagine an automation, HR, automatic notifications, alerts, ...

maybe something like

Code: 
 APPLY_SEND_LIMIT = ${if forany{$recipients_list} { and{ {!match_domain{${domain:$item}}{+local_domains}} {!eqi{${domain:$item}}{${domain:$sender_address}}} } }}

(i don't think it's correct but to give an example here too)

where
@domain.tld to @domain.tld it (not counted)
@domain.tld to @domain1.tld (counted also on same server)

as @sysdev say

his is imho safer regarding spam-runs and hacked mail accounts (1000 mail spike anomalies to internal accounts should be visible, but is less worse than 1000 spammails going external). But counting is still good, even for internal mails. After all it's a service on your server and you want to see numbers.
Click to expand...
 
It is quite easy to think of a synthetic situation when there are a lot of locally sent emails. It would be great to have a real use case where the simple email counting (all sent emails) cannot work.

Of course, I understand that the change in behaviour (starting to count all sent emails, counting at the ACL stage not at the routing stage) is causing problems (users that did not used to reach the limits are starting to reach them). We expected that in most cases, the course of action by the server administrator will be to adjust (increase) the limits. Some users with mostly outbound emails are fine with existing limits. Other users with 50/50 distribution in internal/external emails would be happy with the limit increased two times. Some might require the limit to be disabled or just set to a high value. No matter the use case, as long as the account is not used for bulk emails, there should be a number of emails that it sends per day.

Our aim is to make the send limit feature easy to understand. Just estimate how many emails you send per day and set it as a limit (maybe slightly higher). The old behaviour (counting only external sends) was much harder to explain. The user would need to estimate how many of the emails that he sends are leaving the server, not how many emails he actually sends, to know how to set the limit.

However, the limit was abused just as a tripwire to catch hacked accounts that start spamming. For users who want the send limit to be like any other resource limit (disk quota, number of domains, etc.), counting all emails makes perfect sense. For users who just want a spamming tripwire, counting local sends is useless (just require setting higher limits and that weakens the spam protection).

Considering all that, it would be great to get some feedback on what is the real motivation for asking to be able to run in the old mode (not counting local-to-local sends). Is it one of:
  • (A) I do not want to go over each server, each user account and each mailbox and adjust the limits. I just want backwards compatibility. Maybe new servers or new accounts will use the new counting mode, but not my old servers.
  • (B) I do not care about the limits as limiting user resources. All I need is just a spamming account detector. That is why I do not care about counting emails that do not leave my server (not even emails between different user accounts on the same server).
  • (C) I have a specific use-case that just does not work with counting all emails. Accounts send massive amounts of local mail, but I do not want to give them "unlimited" send limit. I want to be able to make exceptions for send counting, for example not count if sender and recipient domain matches.
  • (D) Something else 😄.
Getting a clearer picture would help us to come up with a better solution. Thanks.
 
We have a customer with a news website with about 30 editors who work in shifts. Most of the mails are initiated by the site itself (rss notifications, articles, stats, changes in hourly plannings, updates etc.), which can easily grow to a total of 500 or more a day depending on what happens. But that website mail account is not supposed to send mail outside it's domain except for a few specific mails. So real outgoing would be set as low as possible.
Why not count and show both?

I don't agree on the part that it's 'abused' as a spamming tripwire. That's exactly what it is supposed to do, imho. It's only use is to prevent spam runs on regular joe's account going apeshit or having the website owner contact you to increase it and taking full responsibility for the mailings he sends.
 
We have clients with over 500 email accounts, and they communicate extensively within their domain. They also have an internal system that sends various notifications to their domain users. However, emails to external domains average around 50-70 per day. We'll have to set a limit of around 10,000 for them; it's currently 500. If something goes wrong, everyone will receive a notification and the emails will be blocked too late.
That's probably why our option is (B).
 
You must log in or register to reply here.
Back
Top