Incorrect FROM Header in DirectAdmin messages

[sparek]

From what I can tell, DirectAdmin is incorrectly setting a From Header address in it's system outgoing email messages.

Specifically I am seeing this in the messages when a new account is created, i.e. the ones with subjects of:

Your account for example.tld is now ready for use.

and

Creator Duplicate: Your account for example.tld is now ready for use.

But I suspect this applies to all outgoing messages being sent by the DirectAdmin panel.

The From Header is being set to whatever the contact email address is for that reseller. The problem arises with DMARC validation at the recipient's end.

If the reseller has their contact email address set to an @yahoo.com email address, then our server (and any other DirectAdmin server) is not authorized to send email from the yahoo.com domain name. This results in the message not being delivered to some recipients.

The From Header should be set to a specific server-only email address. I would propose the same email address that the envelope-sender is set to: diradmin@%the_server_hostname%.

The message does include a Reply-To header which can be set to the reseller's contact email address as that does not play a role in DMARC and SPF validation.

If this is configurable some where in DirectAdmin - I don't know where that is, and if someone can clue me in on where this can be configured - such as a template file - it'd be much appreciated.

I also suspect that this same issue affects other messages that are sent out from DirectAdmin but I don't have an exhaustive list (account suspensions and unsuspensions? Account deletions?)

 

[DrWizzle]

All the templates are located in /usr/local/directadmin/data/templates.

 

[sparek]

I don't think those are message templates. Those look to be more configuration templates (i.e. what actually gets added as a VirtualHost when you create a new account).

Nothing comes up when I search specifically for this new account message:

# grep -lr 'is now ready for use' /usr/local/directadmin/data/templates | wc -l
0

Still, if I'm overlooking something in there, I'm opened to having that pointed out to me.

I suspect this message is set internally in DirectAdmin and would require DirectAdmin to have the system adjusted. But that's just my suspicion.

 

[DrWizzle]

I don't think those are message templates. Those look to be more configuration templates (i.e. what actually gets added as a VirtualHost when you create a new account).

Nothing comes up when I search specifically for this new account message:

# grep -lr 'is now ready for use' /usr/local/directadmin/data/templates | wc -l
0

Still, if I'm overlooking something in there, I'm opened to having that pointed out to me.

I suspect this message is set internally in DirectAdmin and would require DirectAdmin to have the system adjusted. But that's just my suspicion.

User, Admin & Reseller email templates are in:

/usr/local/directadmin/data/users/[admin-username]/u_welcome.txt
/usr/local/directadmin/data/admin/a_welcome.txt
/usr/local/directadmin/data/admin/r_welcome.txt

root@v9 /usr/local/directadmin # cat ./data/users/[admin-username]/u_welcome.txt
Dear Customer,

        Thank you for choosing our service to meet your web hosting needs.

Your account has been created with the following details:

Username:       |username|
Password:       |password|
Domain:         |domain|

To log in immediately, follow this link, using your username and password:

http://|ip|:|PORT|

Once your domain resolves, you will be able to follow this link:

http://www.|domain|:|PORT|

Bandwidth:      |bandwidth| Megabytes
Disk Space:     |quota| Megabytes

Virtual Domains:        |vdomains|
Subdomains:     |nsubdomains|

POP Email Accounts:     |nemails|
Email Forwarders:       |nemailf|
Email Autoresponders:   |nemailr|
Email Mailing Lists:    |nemailml|
POP Server:     mail.|domain|
SMTP Server:    mail.|domain|
Login:  |username|
Password:       |password|

FTP accounts:   |ftp|
Anonymous FTP:  |aftp|
FTP Server:     ftp.|domain|
Login:  |username|
Password:       |password|

IP:     |ip|
Use |ip||*if OWNED!="yes"|/~|username||*endif| to access it until the domain resolves.

You must use these dns servers for your domain. They can be changed through your domain registrar.

NS1:    |ns1|
NS1 IP: |ns1ip|
NS2:    |ns2|
NS2 IP: |ns2ip|

MySQL Databases:        |mysql|
Domain Pointers:        |domainptr|
SSH Access:     |ssh|
Secure Socket Layer:    |ssl|
CGI:    |cgi|
PHP:    |php|
DNS control:    |dnscontrol|

Once again, thank you for choosing our hosting service
Please don't hesitate to contact us if you have any questions

Not sure where the conf file is for the nuts and bolts of mail sending from DA are

 

[sparek]

That appears to be the template for the message. But it doesn't include any of the header information, which is what needs to be modified.

That's why I don't think the header information is customizable. And this would need to be fixed by the actual DirectAdmin developers.

But I'm also a little shocked that this hasn't been brought up before. And that's what gives me pause into thinking there's a setting some where that I'm just overlooking.

 

[Richard G]

Yes it causes also other issues like this one.

It needs to be adressed by DA.

 

[sparek]

This would seem to ultimately be the issue you are having from the post you linked to.

For the longest time, sender spoofing was allowed. And technically there's nothing in the SMTP RFCs that prevent sender spoofing.

When using SMTP, you can use a From address (envelope-sender or header from) of anything. The SMTP server will accept it (save for DirectAdmin's recent "fix" to prevent this). But the validation of From addressing is being done outside of SMTP with SPF, DMARC, and to some extent DKIM.

What this means is that there are A LOT of systems that allow any From address to be used. I've run into this issue with various WordPress plugins that are doing this. It's taken a lot of the major email players - such as Gmail and Yahoo - to enforce strict validation to catch these instances.

No longer can you just use whatever email address as your From address when sending an email. The server your sending the mail out through has to be authorized to send out mail from the domain name used in your From address (again, envelope-sender or header from). No DirectAdmin server or any shared hosting server is going to be authorized to send out mail from a yahoo or gmail email address.

Too many times these applications or plugins are using the user's contact email address as the From address and sending it To the same email address. That's definitely not going to work if you are using an off-server email address.

Unfortunately... best I can tell... we're just forced to wait on the developers (DirectAdmin / WordPress plugin developers) to come to their senses and fix these issues.

 

[DrWizzle]

What would be nice is if the welcome message could be sent out and a proper name be on the email, like the emails sent out by WHMCS. Doesn't show a professional look if an email drops into your junk box from "[admin-account-name]" with login data.

What would also be nice is if you could create some HTML emails.

 

[sparek]

Meh, I'd just settle for the correct usage of the From header right now.

If a reseller doesn't like the From header being from a generic name, then they can uncheck the checkbox that sends an email.

The problem right now is that resellers are oblivious to the fact that this From header is being used incorrectly. And don't understand that their messages aren't going out or that it's causing the server to be treated as a spam source.

 

[sparek]

You can hackify your way to a solution to this by using the sendmail_pre custom hook.

Create the file /usr/local/directadmin/scripts/custom/sendmail_pre.sh with the contents:

#!/bin/bash

full_message=$(echo "${full_message}" | sed -E "s/^[[:space:]]*[Ff][Rr][Oo][Mm][[:space:]]*:[[:space:]]*.*$/From: <diradmin@${HOSTNAME}>/")

echo "${full_message}" | /usr/sbin/sendmail -t -i -f diradmin

This effectively replaces the DirectAdmin system for sending emails (emails sent from within the DirectAdmin panel).

The From: header is completely replaced with From: <diradmin@${HOSTNAME}>

The new ${full_message} with the updated From header is then echo'd and pipe to /usr/sbin/sendmail -t -i -f diradmin

The -t flag tells sendmail to read the message as if it includes headers.

The -i flag tells it to ignore periods (.) on a line by themselves.

And the -f diradmin tells sendmail to use an envelope sender of diradmin - which will automatically get the server's hostname attached to it... diradmin@${HOSTNAME}

Not saying I love this solution. Mainly because I don't know what all messages sendmail_pre refers to. My preference would still be for DirectAdmin to fix the system within their code. But if you are desperate for a solution, glean from this what you will.

 

[zEitEr]

Create the file /usr/local/directadmin/scripts/custom/sendmail_pre.sh with the contents:

Thank you for the script. I believe it won't prevent an original attempt of Directadmin to send an email? DirectAdmin will still try and send an email on its own, and another copy will be sent with the hook. Correct?

 

[sparek]

Thank you for the script. I believe it won't prevent an original attempt of Directadmin to send an email? DirectAdmin will still try and send an email on its own, and another copy will be sent with the hook. Correct?

That's what I thought, but wasn't what I was seeing.

Then I found the documentation for this hook:

Miscellaneous Hooks | Directadmin Docs

DirectAdmin Knowledge Base

docs.directadmin.com

And according to that documentation, "This script will allow you to override the sendmail call and handle the message yourself."

That's why you have to pipe the new ${full_message} through sendmail (or send it however you want to send it).

So it would appear that if /usr/local/directadmin/scripts/custom/sendmail_pre.sh exists, then DirectAdmin's own mailing system won't send an email. Regardless of whether or not if /usr/local/directadmin/scripts/custom/sendmail_pre.sh is actually sending the message.

Feel free to try it yourself and come to your own conclusions. But that is what I am seeing.

 

[zEitEr]

OK, thank you. I will try it on my own as well

 

[gate2vn]

I just saw the same problem when one account within a reseller is using over its limitation. Like you said, sending to Yahoo will be blocked.

 

[Richard G]

The From: header is completely replaced with From: <diradmin@${HOSTNAME}>

Do I understand correctly that it will replace the [email protected] with diradmin@hostname or am I mistaken?
But if this is the case, won't there be an issue anymore on the receiving part like Gmail due to the fact that maybe SPF, DKIM and DMARC of resellerdomain is used? Or will that be skipped?

Still.... the issue should be fixed bij DA devs indeed. Maybe @fln can have a look at this issue? Because external systems like gMail etc. get more strict ever time.

 

[DrWizzle]

Do I understand correctly that it will replace the [email protected] with diradmin@hostname or am I mistaken?
But if this is the case, won't there be an issue anymore on the receiving part like Gmail due to the fact that maybe SPF, DKIM and DMARC of resellerdomain is used? Or will that be skipped?

Still.... the issue should be fixed bij DA devs indeed. Maybe @fln can have a look at this issue? Because external systems like gMail etc. get more strict ever time.

Out of curiousity, before adding that pre hook above to test, I tested Hotmail and GMail. The mail dropped straight into the inbox with everything passing. I've heard of a lot of people's mail going to spam or not at all. Here's the gmail header without the sendmail_pre.sh being created yet.

I'll create a yahoo account in a bit and test the deliverability.

 

[sparek]

All of this rears it's head when you Add a New User and have the checkbox for Send E-Mail Notification checked.

I suspect this is done in other emails sent out by DirectAdmin as well. I suspect this is the case when an account reaches it's disk quota limit or bandwidth limit. I see some evidence of that, but can't verify that. But creating an account is easy enough to see and duplicate this issue.

If you check that checkbox and proceed to Add a New User, then an email message will be sent out.

The From header of that email is going to be set to whatever Contact E-Mail Address is set for the user that created the account (whether that be a reseller or the main admin account).

You can see this Contact E-Mail Address by clicking the Hello, %username% drop down box in the top right corner in the DirectAdmin panel (for a reseller or the main admin account) and choosing Profile and noting the Contact E-Mail Address.

Without this sendmail_pre.sh modification, when adding a new user the welcome email will be sent to the email address specified in the E-mail field. That message will use a From header email address that is the same as the creator's Contact E-Mail Address.

This is wrong.

If the creator's Contact E-Mail Address is a yahoo, gmail, or some other off-server email address then chances are great your DirectAdmin server (the one you just added a new user to) will not be authorized (as in having the server's IP listed in the From header's email address's domain name's SPF record) to send out email from that Contact E-Mail Address.

What this sendmail_pre.sh script does, it overwrites the WHOLE mechanism that is sending out these emails. I suspect the mechanism that is sending out these emails is buried in the directadmin binary (that's why it will take DirectAdmin's developers to fix the issue). So if /usr/local/directadmin/scripts/custom/sendmail_pre.sh exists, that whole mechanism defers to to /usr/local/directadmin/scripts/custom/sendmail_pre.sh to send out any emails (regardless of whether /usr/local/directadmin/scripts/custom/sendmail_pre.sh actually does that).

The modification I have listed in the sendmail_pre.sh script takes the From header in these messages - no matter what it is set to - and replaces it with diradmin@${HOSTNAME} - which is the same from address that the envelope-sender is set to.

I thought I had a write up some where that explains the difference between the envelope-sender (sometimes called a return-path) and a From header, but I can't find it. So I'll use a little space here to explain the difference:

When sending an email over SMTP the basic commands sent are:

EHLO servername
MAIL FROM: <[email protected]>
RCPT TO: <[email protected]>
DATA
.
.
.

I refer to this as an SMTP transaction - I'm not sure if that is the proper title, but that's what I'm going with.

You'll notice... there is no Subject listed here. That's because the Subject is a header. It's listed in the DATA part of the transaction. If you look at a full transaction:

EHLO servername
MAIL FROM: <[email protected]>
RCPT TO: <[email protected]>
DATA
From: <[email protected]>
To: <[email protected]>
Subject: Test Message

This is my test message
.
QUIT

In this example:

MAIL FROM: <[email protected]>

is the envelope-sender or return-path. This is what the MTA (Exim, Sendmail, Postfix, etc) sees when it receives the message.

By contrast:

From: <[email protected]>

is the From header. The From header - and most any header - is what the MUA (emai client, Outlook, Thunderbird, etc) will show.

Note that these don't match. From a pure SMTP point of view... this is valid. There's nothing in the SMTP spec that says the envelope-sender and From header have to match. There's nothing in the SMTP spec that says a From header has to exist.

But when it comes to validating an email at the recipient's end, that's where SPF and DMARC (and DKIM) come into play. Now, technically SPF is supposed to check the domain in the envelope-sender and see if the sending server IP is authorized to send out mail from the domain listed in the envelope-sender. I can't vouch and say that is always the case. DMARC comes around and wants to align the envelope-sender with the From Header. DMARC says: "If the SPF for the envelope-sender passes, then does the domain in the envelope-sender match the domain in the From header?" If both of these statements are true, then the SPF portion of DMARC passes. If neither one is true (i.e. SPF fails for the envelope-sender OR the domain in the envelope-sender and the From header are not matched) then the SPF portion of DMARC fails. Technically you still have the DKIM portion of DMARC that can succeed, but this really depends on how strictly a recipient server is actually following the DMARC specification.

Now back to the original topic...

So what this sendmail_pre.sh script is doing is setting the From header to diradmin@%hostname% - which, at least in all that I have seen - is the email address used as the envelope-sender in these emails. So DMARC SPF alignment will match. Now... %hostname% still has to have a valid SPF record that includes the server's IP address as being a valid sender for the %hostname% domain - but I assume that is already done. If that's not done, then SPF is going to fail anyway, so it really doesn't matter if the envelope-sender and From header are in alignment.

The sendmail_pre.sh is taking whatever is set as the From header and replacing it completely with diradmin@${HOSTNAME}. It still maintains the creator's Contact E-Mail Address as the Reply-To header, so if the recipient replies to the message the message SHOULD be sent to the creator's Contact E-Mail Address (assuming the client's MUA reacts to the Reply-To header correctly).

I would argue that the DirectAdmin developers should modify the mechanism that sends out these emails and always use diradmin@${HOSTNAME} as the From header and thereby negating the need for this "hack" in the sendmail_pre.sh script. But without knowing how long that will take to reach the developers, using the sendmail_pre.sh script SHOULD provide a patch-work to fix this issue.

 

[sparek]

Out of curiousity, before adding that pre hook above to test, I tested Hotmail and GMail. The mail dropped straight into the inbox with everything passing. I've heard of a lot of people's mail going to spam or not at all. Here's the gmail header without the sendmail_pre.sh being created yet.

View attachment 9443

I'll create a yahoo account in a bit and test the deliverability.

Is %%%%%%%%%.co.uk a domain that is setup on the same server you are creating accounts on?

I'm guessing that is the case and even though SPF alignment isn't passing with DMARC, DKIM alignment is.

It's when the From header is using a foreign domain name - one that does not exist and resolve to your server - that all of this alignment fails.

If the From header is using a yahoo.com email address... then the message won't be signed by Yahoo's DKIM key (you don't have access to this private key so you can't correctly sign it as being from Yahoo). So for one, it won't include a DKIM signature for the yahoo.com domain name which fails DKIM automatically. And even if it did... it won't match the signature of the DKIM public key stored in yahoo.com's DNS record, so it will fail.

 

[DrWizzle]

Is %%%%%%%%%.co.uk a domain that is setup on the same server you are creating accounts on?

I'm guessing that is the case and even though SPF alignment isn't passing with DMARC, DKIM alignment is.

It's when the From header is using a foreign domain name - one that does not exist and resolve to your server - that all of this alignment fails.

If the From header is using a yahoo.com email address... then the message won't be signed by Yahoo's DKIM key (you don't have access to this private key so you can't correctly sign it as being from Yahoo). So for one, it won't include a DKIM signature for the yahoo.com domain name which fails DKIM automatically. And even if it did... it won't match the signature of the DKIM public key stored in yahoo.com's DNS record, so it will fail.

Yes, that domain is used solely for server hostnames, nameservers and their dns server's hostnames. Every server will have that domain in its structure. I've created that domain as an account on main server also so I can use email (simply for sending out messages) so i've got a DKIM, SPF and DMARC record for it to try and make things as deliverable as possible.

This email stuff is a fine art and I'm still learning about things 20 years on!

I get that if someone used a yahoo address or other in their header nothing will match, (I hope I've read what you've wrote correctly) and I wonder why anyone would use an email address that isn't linked to their server in some way as the built in mail services are free and they would have a domain for it anyways?

I do hope these bugs you all mention though are fixed as I think it's imperative you get messages out to your customers, whether its provision, suspension or even limits. They need to know as well as you what's going on.

 

[DrWizzle]

The sendmail_pre.sh is taking whatever is set as the From header and replacing it completely with diradmin@${HOSTNAME}. It still maintains the creator's Contact E-Mail Address as the Reply-To header, so if the recipient replies to the message the message SHOULD be sent to the creator's Contact E-Mail Address (assuming the client's MUA reacts to the Reply-To header correctly).

I tried out that script, out of curiosity, and i'm guessing it would work if the {hostname} variable actually provided the domain. Not sure if it's my setup but the hostname is appended to the new header without the domain, even though my hostname is set in the admin panel as a FQDN.

What I mean here is this, with that script as is, my hostname for the email is malformed and fails SPF, DKIM and DMARC so lands in the Spam:

I'm using Ubuntu 24.04 and the latest release of DA.