I finally got a response from Barracuda that they were going to un-blacklist me, and check me out again in 30 days. The new server is no longer blacklisted. The problem is, I'm not sure what to do different over the next 30 days. MXToolbox likes my E-Mail server. I did register my IPs for their free RBL service, which for now I had Exim drop them. I did prove with an nslookup that the server was indeed using the BIND installed on my server, and resolv.conf only has our IPs that point to the local DNS server, and there is no 127.0.0.1 so it cannot route through localhost and thereby be a violation of the rules with Barracuda.
During this period I've had to spend WAY too much time thinking about E-Mails and protocols. We forward E-Mails consciously from time to time, but many set up forwarders, and about half of them are to external E-Mail accounts like Google, Yahoo!, Hotmail.com/Outlook.com. Whenever you forward an message, you have to preserve the original sender as well as the receiver so they can respond to it. So how would that work out?
I. SPF:
A. Someones E-Mail account gets compromised and they are broadcasting spam. It hits your E-Mail server. Your server does a DNS lookup on the senders domain for the SPF record, and reads it. Your server sees that envelope-from address and the IP address match, so you drop the score of the E-Mail and accept it. It does not consider that the From the user sees and the one in the envelope from do not match.
B. Your E-Mail server sees that it needs to forward the message to GMail. The forwarding email server, your server, must impersonate the original email server in order to preserve the original sender information. The GMail server reads the envelope-from, does a DNS lookup on the senders domain for the SPF record, and reads it. The destination server realizes that your E-Mail server is trying to deliver E-Mail it is not authorized to send E-Mails for the sending domain, and it naturally knows your server's IP address. Now who is the scammer/spammer? Not the server with the hacked account that can pass the SPF test, it is yours because you cannot. In fact you've been passing lots of E-Mails that your server hasn't been authorized to pass (legitimately). The only time SPF forwards will NOT fail is if they are forwarded to another user on the same physical E-Mail server.
C. GMail tells RBLs that you are acting like a spammer. Now let's add to that some reports from Yahoo! and Hotmail/Outlook.com, Office365, etc. where you forward E-Mails.
II. DKIM:
A. Someones E-Mail account gets compromised. The hacker sends messages from that account only this time the server adds a DKIM signature. One thing that is different is the From: field that the user sees is the header From: filed, the one that the user sees when he makes the E-Mail, that is always included in the hash it creates, while SPF uses the (envelope-from field, something the user never sees. It actually cannot happen any other way because at the point where the hashing and encryption have to take place, you couldn't include the information from the envelopes because they don't exist yet. The destination E-Mail server sees sees that it has a DKIM signature, which reveals which “domain/selector” combination used in the encryption process. To validate the signature, the destination E-Mail server will run a DNS query to find the DKIM public key contained in a TXT record for that domain on your server. With that key it can decrypt it back to the original hash. The destination E-Mail server then takes the elements of the email signed by DKIM and generates its own hash of these elements, and compares it to the decrypted hash from the DKIM signature. If they match, we know that:
1. The DKIM domain really does “own” the email, otherwise the decryption process wouldn’t have worked in the first place.
2. The elements of the email signed by DKIM were not changed in transit (if they were changed, the hashes would not match)
If it all validates and your server drops the spam score.
B. Your E-Mail server sees that it needs to forward the message to GMail. As before, the forwarding email server, your server, must impersonate the original email server in order to preserve the original sender information so the recipient can respond to the original sender. However, things play out a little differently this time. GMail's E-Mail server notices the message has a DKIM signature, so it skips the SPF test because it knows it fails half the time because of how often people forward E-Mail messages. DKIM doesn't care how the message got there, it does the same thing your server did, and looks for the DKIM TXT record in the DNS that originated with the E-Mail, and reads the public key. If after it unencrypts the signature and validates the hash, it validates so GMail's server drops the spam score.
C. GMail finds out the message is spam, but with DKIM, they know the message originated from the domain listed in the message header From:, not your E-Mail server. It doesn't mean you are completely off the hook because you could have tampered with the content, but they do know it didn't originate from your server. It's more likely to report to RBLs that the spam is coming from domain in the header From: than from you, the forwarding domain.
*Reality. Anyone can write or changing anything in the message that is not in the hash, and it will will not mess up the DKIM signature. Because DKIM only signs the specified parts of the message, the message can be forwarded on by an intermediary that inserts the extra fields, and the signature will still match. This is called a replay attack. You might believe that if you have DKIM encrypt more of the message, the safer the messages would be. The problem with that scenario is that Anti-Malware and Anti-Spam software often write changes to the E-Mail, and if one of the fields is in the hash, then DKIM sees it as a message that has been tampered with. Another is the myth that DKIM cannot be spoofed. What hackers do is add a second From:, To:, or Subject:.
III. DMARC
DMARC can be used with SPF, or DKIM, or both. DMARC's function is simple
A. For SPF it is matching the header "From:" domain name portion that the user sees, with the envelope's "(envelope from” domain name that the SPF check uses. (alignment)
B. For DKIM, it is about matching the header "From:” domain name that the user sees, with the “d= domain name” in the DKIM signature. (alignment) A mismatch there or any of the other fields in the hash indicates the message has been tampered with.
C. To pass DMARC authentication, a message must pass SPF authentication and SPF alignment and/or DKIM authentication and DKIM alignment. For DMARC to use SPF and DKIM together to lower spam scores significantly and increase trust against spoofing, they must be aligned together. E.G. both using the same domain. A message will fail DMARC and often acceptance if the sending and destination domains support both SPF and DMARC and fails both.
Reality: Dave Rand and Doug Otis of Trend Micro argue that a weakness in the specification means that end-users can be effectively spoofed if they don't check for duplicate header records and since they are in there anyway, that should be checked and by messages that pass both DMARC testseven with the criteria of SPF, DKIM, and DMARC being satisfied and that it would be trivial to add code to check for duplicate headers. The committee argued that would be a malformed E-Mail, and that it should not be handled by DMARC.
Summary Thoughts
- SPF is almost universal, and drop-dead simple to deploy. While the destination E-Mail server can determine from it if the E-Mail server is authorized to deliver E-Mail for the sending domain, it cannot use it to raise the spam score if it doesn't match because every E-Mail that is forwarded meets that criteria. Shucks, I could set up an E-Mail server with an spf record and domain spam-inc.com, send spam to people, and have the E-Mail message they receive look like it come from you to them, because spf simply verifies the (envelope-from, which would be spam-inc.com and I have a matching spf record to match.
- DKIM was made to "solve" the spoofing problem by hashing and encrypting the header From: field ans several others the sending user sees and created and should not change during its travel to its destination. When the destination E-Mail server sees the DKIM code in the message, it can use the header From: to find the domain's public key to unencrypt the hash with, and compare that hash to the hash that it generates from the same fields in the current version of the message to determine if any of them have been tampered with. It isn't interested in how the message got there. The problem is we know that it can be spoofed by simply adding additional headers.
- If you add DMARC to SPF, DMARC looks at both the (envelope-from that the user does not see, and the header From:, which the user does see, and checks for a match. As stated before, SPF fails if a message is simply forwarded, so you can't really raise the spam score, but you could drop it it some.
- If you add DMARC to DKIM, DMARC looks at both the (envelope-from that the user does not see, and the DKIM's secure hash that includes -d domain, and the From:, which the user does see, and checks for a match.
- If both SPF and DKIM succeed, it means you have a message where the important fields have not changed since it was accepted by its E-Mail server, and the destination server can be sure it is talking to one that the domain authorized to send E-Mail for it. DMARC does not by itself deliver on its design goal to prevent spoofing due to the multiple header field vulnerability.
- It is my opinion that the whole DMARC strategy is so flawed that I struggle to find any real value. To get a true picture of how easy it is for a spammer or scammer to get around DMARC and even use it to his advantage, see
Breaking DKIM - on Purpose and by Chance. These protocols are so short-sighted it's as though the Bank of Nigeria and ransomware authors staffed the committees that came up with them.
- It seems like the old school technologies of Bayesian filtering, and clever, highly responsive, Anti-Virus and Anti-Spam to minimize the damage, are the most effective. RBLs, due to their nature, need to be slow to react to prevent false positives and thereby turn into a nuisance.