Be aware of the Google/Yahoo requirements for mail by february 2024

[Richard G]

I don't know if this is already known by everybody, but just to be sure, be aware of the requirements by Google and Yahoo as announced in October last year.

For Google, if your business sends more than 5,000 messages to Gmail addresses, you must adopt SPF, DKIM, and DMARC by February 2024. Yahoo will apply the same trio of requirements to “bulk senders” in the first quarter of 2024, though they have not defined what constitutes a bulk sender.

The new email requirements include both SPF and DKIM records for authenticating email-sending domains, a DMARC record for the domain, and a “From” header that matches either the SPF or DMARC record, known as “alignment.” In addition, marketers must keep spam rates below 0.3% and provide the ability to unsubscribe with a single click if the recipient chooses.

Here it's a bit more specific:

• Authenticate all messages with DMARC (technically, authenticate all messages with SPF or DKIM aligned with the From domain)
• Send from a domain with a DMARC policy of at least p=none
• Have valid forward and reverse DNS that match each other
• Use the one-click unsubscribe header and an unsubscribe link in the footer
• Maintain a low spam rate of < 0.1%
• Encrypt your email (technically, require TLS)

DA does enable SPF and DKIM by default. If you run your own nameservers on the DA server, change the DKIM setting in directadmin.conf to DKIM=1 so it will be created automatically on new domain creation.
For the DMARC record you can copy the dns_txt.conf from:
/usr/local/directadmin/data/templates
to the
/usr/local/directadmin/data/templates/custom/ directory.
Create the /custom/ directory there yourself in case it might not exist.

Then add this line in the dns_txt.conf file in the custom directory:
_dmarc="v=DMARC1; p=none;"
Then it will be added when a new domain is created.

Use the docs to find out how to massive change existing DNS records.

Might be good if DA could build in a (maybe switchable) option in directadmin.conf to have this enabled or disabled by default so we don't need the custom conf anymore.

Be aware that if you run external namservers like Cloudflare or Vultr or others, you have to copy your SPF, DKIM and DMARC records to these nameservers or use the LEGO option from DA to do it automatically if possible.

 

[sparek]

Someone can correct me if I'm wrong, but I believe this means forwarding mail to a Google email address will no longer work.

Forwarding mail always breaks SPF. Because the server IP address sending the message to Google (presumably your server), will not be in the acceptable IP address list for the original email address.

If someone from a Yahoo address sends a message to your email address hosted on your server and that email address forwards the message to Google, then from Google's perspective: It sees your server sending an email to them from a Yahoo address. Your server's IP address won't be listed in Yahoo's SPF record.

I don't really have an issue with this. I've long argued that external email forwarders (forwarding mail off of the server) is a horrible idea and something that needs to be abolished. Perhaps this move by Google will aid in this.

A better solution - at least with Google - is to set your Gmail account to retrieve mail from your server via a POP3 email account.

Someone from Yahoo sends a message to an email address hosted on your server, make that needs to be a POP3 account. Then in your Gmail settings configure it to log into your server via POP3 with the username and password credentials of that POP3 account and retrieve messages - including the one that someone from Yahoo sent.

 

[Richard G]

As far as I know it will still work with forwards as the DKIM still has the signature of the original sender, just as forwards work now too. Why should there be any difference?
I don't think these new rules will affect normal users either. It says more than 5000 mails.

A better solution - at least with Google - is to set your Gmail account to retrieve mail from your server via a POP3 email account.

Out of privacy I wouldn't advise this. Also this is a choice a user can make. A normal user does not need to do anything, next to that you don't want Google to know all your stuff. IMHO best solution is still a normal e-mail client as for normal users nothing will change.

There are just 2 things required for server owners if they send more than 5K a month.
Just put the DMARC in the DNS records and take care that customers with large mail sending amounts will have the unsubscribe link in their mails too.

For the rest, no need to make things complicated as far as I can see.

 

[Ohm J]

this affect only small company doesn't have IP Rotating ( Load Balance ). So even they have all dkim, dmarc, spf. It could leads to spam rate ( 0.
1% ).

So small company that's still use to make ads on email, Will got big impact from this change.

 

[sparek]

As far as I know it will still work with forwards as the DKIM still has the signature of the original sender, just as forwards work now too. Why should there be any difference?

Forwarding mail always breaks SPF. Now if the Google server (or whatever the external destination is) doesn't care about SPF and only looks at DKIM, then maybe you can get by. But if you automatically forward any email to an external email server, it's always going to fail SPF.

Out of privacy I wouldn't advise this. Also this is a choice a user can make. A normal user does not need to do anything, next to that you don't want Google to know all your stuff. IMHO best solution is still a normal e-mail client as for normal users nothing will change.

What's the privacy concern?

If you forward your mail to Google - Google gets to read your emails.

If you POP mail from your server to Google - Google gets to read your emails.

If you're concerned about sharing a password for your email account with Google... I'd have to question why that is a concern. Are you reusing the same password for everything, including this email account? Don't do that.

Like I said, I'm not a fan of external email forwarders. I've been on this soapbox for 20 years. When people complain about all the spam they receive, well this is why. SPF was invented so that email from a domain could be verified as authenticate given a list of IP addresses that the domain will be sending from. But because the community would not give up their desire to have automatic external forwarders this nullified the advantage that SPF gave. So instead of having explicit "mail for this domain should only comes from these IP addresses" we have "mail from this domain should come from these IP addresses, but if it doesn't that might be OK too, so better to just let the message come through from any IP address"

If you want all of your mail to be in your Gmail account - and I don't have an issue with this - the ability to POP mail into your Gmail account is solid. Since there's no MTA traversing and because the Gmail user has to explicitly PULL these messages from the server, spam messages retrieved in this manner can't be considered spam from your server. If the concern is that you're giving Google the username and password to your email account... just make sure that password is unique. Are you concerned that Google has your Gmail password too?

 

[exlhost]

Running these requirements already for years. You should run these to build a good email delivery reputation.

 

[Erulezz]

Test your settings on;

Newsletters spam test by mail-tester.com

mail-tester.com is a free online service that allows you to test your emails for Spam, Malformed Content and Mail Server Configuration problems

www.mail-tester.com

 

[Richard G]

But if you automatically forward any email to an external email server, it's always going to fail SPF.

Agreed, but that doesn't need to be an issue or a new issue due to DKIM.
If it wasn't an issue in the past neither will it in the future with the new rules. It's not about receiving mail, it's about sending loads of mail (more than 5000) to gmail. Unless all your customers use gmail for their domains.

If you want to advise your users to have Google pickup their domain mail, it's a free choice ofcourse. You're correct that this is better than forwarding if most of your customers is doing this.

On our servers this is not the case, they send mail to other gmail accounts, only a few forward their mail. So it's important to have the correct settings anyway. Which we already did last year to be sure.

But it's everybody's free choice on how to do things.
I just wanted to make this known to those who were not aware of this new policy yet.

 

[equus]

If you're forwarding emails from your own server, include your server IP in your SPF record.

 

[Stije]

Then add this line in the dns_txt.conf file in the custom directory:
_dmarc="v=DMARC1; p=none;"

Why do you use

v=DMARC1; p=none;

Any reason?

I would suggest

"v=DMARC1; p=reject; sp=none;"

or maybe

"v=DMARC1; p=quarantine; sp=none; rua=mailto:[email protected]"

If set to none then DMARC verification, if failed, is still delivered to normal inbox. Which not really does anything so why set it up at all then.

Also nice to do by the way is create an spf record with all ip's you send mail from and call it for example spf.example.com and set it with DA on your servers:

/usr/local/directadmin/directadmin config-set extra_spf_value "include:spf.example.com"

This will create a nice spf record instead of the ip's and you just add ip's to the main spf record without having to worry about customers with wrong spf.

 

[Richard G]

Any reason?

Yes. The reason is to precent any issues which might rise for the customer. And to keep things easy as most customers don't know what they have to do with it.

If you use the p=reject; in there as you suggest, chances are mail will get into spamboxes or not being delivered. Some odd things can happen with forwarders and especially with auto-reply and vacation messages as you can see on the forums.

And why adding a non existing rua address?

So that's why I gave my example that way, so they will work for everybody, no matter what the customer has configured. And they can be adjusted afterwards if needed.

Which not really does anything so why set it up at all then.

Just and only to fullfill the requirement of Google and Yahoo to have a DMARC record for systems sending over 5K mail a month. That's why.
So this is a quick fix and things can be changed afterwards as far as needed. For most systems a correct SPF and DKIM is enough, so a working DMARC isn't really needed.
And if you setup DMARC, either let it do nothing, or set it up correctly. And that can not be done for everybody. Because you best then setup a correct policy and rua and ruf addresses so it can be checked everything is working correctly.
Most customers don't even have clue about these things. So this is a quick fix to mee requirements. It's up to the various hosts or their customers to see what they are going to do with it.
For loads of systems it will not even be required as they don't reach the 5K mail/month to Gmail or yahoo.

Also nice to do by the way is create an spf record with all ip's you send mail from

That's a nice suggestion for domains which sends from multiple ip's. Like Microsoft or Ziggo. Most customers are not that big and they don't have to worry about a wrong SPF anyway with the SPF which is set by DA itself.

 

[Stije]

And why adding a non existing rua address?

It is an example addres as I don't know the domain in question off course haha

*edit for clear reading

Yes. The reason is to precent any issues which might rise for the customer. And to keep things easy as most customers don't know what they have to do with

True But sometimes it also good to give them some learning moments. It is also good for admins to set up a good system off course. But I totally understand

If you use the p=reject; in there as you suggest, chances are mail will get into spamboxes or not being delivered. Some odd things can happen with forwarders and especially with auto-reply and vacation messages as you can see on the forums.

This can be fixed as also seen on the fora. Reject is not delivered in spam or junk folders. It get's rejected directly which is the intention but yes with forwards it can be annoying.

That is why quarantine is pretty solid for still having valid mail sent to the inbox and not valid will be sent to junk/spam. Also you could use it like this:

v=DMARC1; p=quarantine; pct=25;

Which means that only 25% of the email messages sent from your domain would be sent to spam if they fail authentication. You can increase these percentages over time and eventually remove it.

 

[Richard G]

It is an example addres as I don't know the domain in question off course haha

So then you can't add it serverwide. And that was the idea, to put in a solution which works serverwide.

This can be fixed as also seen on the fora. Reject is not delivered in spam or junk folders. It get's rejected directly which is the intention but yes with forwards it can be annoying.

Exactly, and rejects can not always be fixed for the forwards. It's something to worry about afterwards if people really want to start using DMARC.
The quarantaine setting will have the same effect, 25% can end up in spam then then you have confusing customers who don't know what is going on.
"Huh? Sometimes my mail gets into spam and sometimes not, what is going on?" And things like that.

And see my last alinea, people tend to forget to copy the records to the external nameservers. There are also customers who choose their own registrar, if they forget to copy it, things get messed up.

So as said, ofcourse there are better solutions. But my solution is one which can be used by everyone, even people not interested in DMARC, and have a good way to meet the requirements.

Ofcourse everybody is free to change the example to their own needs.
I've seen somebody suggestion for example to put in something like rua=mailto:rua@|DOMAIN| if I'm correct, which would create the rua mail address for the domain the dns record is from.
However, in that case, one has to teach his users to create and how to work with that mail address, because it's nonsense to have on in there and don't use it.

So there is my quick and dirty solution which can be used on every server for everybody, which works for everybody and does not get quarantained or rejected in any way.
Because a lot of admins don't to the customisation to get rid of the autoreply issue for example (I didn't either, at least not yet).

But my example is still is an example. So ofcourse one can change this to one's needs before or after creation.
Only if I would give all kinds of examples, my message would get too long (and it's long anyway). So I've limited myself to a working example, not doing anything, so not able to disturb anything, but meet requirements.

 

[Stije]

So then you can't add it serverwide. And that was the idea, to put in a solution which works serverwide.

Use

rua=mailto:dmarc@|DOMAIN|

for in dns_txt.conf and create default email via domain_create_post.sh

Or use 1 address for the serveradmin if users don't know what they are doing so serveradmin can respond and fix or notify customer. I agree on the mindset it is easier but it also makes it easier to send non valid mail and customer can get blocked because of this. It is always a bit of a struggle yes.

 

[Richard G]

for in dns_txt.conf and create default email via domain_create_post.sh

Correct, which takes extra work, because you have to contact all users again.
These are all nice solutions to customize if you know what you want, and if you want to give all your users new mail addresses to work with.

Loads of host don't, hence I don't use it in the example. But yes, this way ofcourse it can be done for those who want.
As said, there are all kids of ideas which can be done.

For the SPF record for example I have changed the ~all to -all to make it strict for everybody.

 

[Stije]

For the SPF record for example I have changed the ~all to -all to make it strict for everybody.

Really so you do not have forwards or something?

Because SPF hard (-all) may cause DKIM to be ignored. Even if DKIM/DMARC is correct it might fail with forwarders for example. If you use DMARC/DKIM best is too use SPF sof (~all) See also RFC's about this 7489 and 7208

Some receiver architectures might implement SPF in advance of any
   DMARC operations.  This means that a "-" prefix on a sender's SPF
   mechanism, such as "-all", could cause that rejection to go into
   effect early in handling, causing message rejection before any DMARC
   processing takes place.  Operators choosing to use "-all" should be
   aware of this.

 

[equus]

Or use 1 address for the serveradmin if users don't know what they are doing so serveradmin can respond and fix or notify customer. I agree on the mindset it is easier but it also makes it easier to send non valid mail and customer can get blocked because of this. It is always a bit of a struggle yes.

The problem with this is then you have DMARC specification (RFC7489) which describes an authorization mechanism that report generators must implement in Section 7.1 for DMARC records outside your domain.

 

[Richard G]

Because SPF hard (-all) may cause DKIM to be ignored.

It says "may" and "might", but I never have seen this happening with my customers. And I only gave this as an example (not as advise!) of what kind of things and idea's could be set in to place when using SPF and DMARC records, showing that it's everybody's own choice as to how they want to configure things.

Terms "may" and "might" are possibilities so they don't need to occur. However it is suggested that for this reason one might better use the softfail to prevent that possible risk. So yes, it might be better to use ~all.
But as said, I was just giving an example (not advise) from all kind of options and I do use the -all option. If I encounter issues it's easy enough with 1 command to set all of them to ~all if needed.

 

[equus]

To be fair, I also prefer the -all mechanism.

 

[thaizone]

Just add all IP addresses in your SPF record. Just make sure it doesn't go over the 255-character limit for TXT records. Also, throw in the "-all" option for added security. And before you go ahead and deploy it, be sure to double-check everything using a web tool.