A client told me that some of their users were not getting their one-time password. The email had arrived, just not in the inbox. It was sitting in their spam or junk folder.
That surprised me, because I thought I had already dealt with this. The OTPs were sent through Cloudflare Email Service, and the domain had SPF, DKIM and DMARC set up. As far as I understood email delivery, those three records were the job. Get them right and the mail lands in the inbox.
That understanding was half right, and the half I had wrong is worth writing down.
What I knew going in
I knew what the three records were for:
- SPF tells Google and other mail providers: this sending server or service is allowed to send email for this domain.
- DKIM adds a cryptographic signature, so a provider can verify the message was signed by the domain and that the signed parts weren't altered on the way.
- DMARC tells providers: if SPF or DKIM don't authenticate, or don't align
with the From domain, follow this policy. The policy is
none,quarantineorreject.
None of that was wrong, and none of it was broken. What was wrong was the conclusion I drew from it.
All three records answer one question: is this email really from the domain it says it is from? I had been treating a yes to that question as a yes to a different one: should this email go in the inbox?
"Some users" was the clue
The report did not say the OTPs were going to spam. It said they were going to spam for some users.
A broken record tends not to behave like that. A DKIM key that does not match fails on every message, and a DMARC policy is applied the same way to every failing message at a provider that enforces it. Mail from the same sender, with the same template, reaching some inboxes and not others points somewhere else: to decisions being made after authentication has already passed.
Those decisions get made in two places, and I had not accounted for either.
Every provider runs its own filter
Authentication is the first thing a mail provider checks, not the last. Once a message passes, Gmail, Outlook, Yahoo and every company mail server still run their own filtering, each in their own way, and none of them share a verdict.
Much of that filtering comes down to reputation:
- Sender reputation is the history of mail from the domain: how much it sends, how consistently, and how often recipients mark it as spam.
- IP reputation is the history of the servers the mail actually leaves from. With a sending service, you may not be the only customer sending from those servers.
Neither of those is something I can set in DNS. A provider that has seen little mail from a domain has little reason to trust it yet, however well that domain authenticates.
What is in the email counts too
The other thing I learned is that filters look at the message itself, not only at who sent it.
An email can carry more than one version of its body. The HTML version is the one that looks good, with the layout, the branding and the code in large type. A plain-text version can travel alongside it in the same message, and the reader's mail app shows whichever one it displays best. Some apps also let the reader switch to plain text if that is what they prefer.
That plain-text version matters to spam filters as well. Some prefer to see a
reasonable text version that matches the HTML. An email that is HTML only, or
one whose text version says something different from its HTML, is the kind of
thing some filters count against a message. SpamAssassin, for example, has a
rule for each: MIME_HTML_ONLY and MPART_ALT_DIFF.
An OTP email is an awkward case even when it is built properly. It is short, it is urgent, and it asks the reader to act on a code, which is also a fair description of a phishing email. That is one more reason not to give a filter anything else to be suspicious of.
And every user runs their own
The part I missed completely is that the person receiving the email has a say as well.
Users train their own spam filters. Someone who has marked similar emails as spam before is teaching their provider to send the next one to junk. People also set up rules, block senders, and use work or school accounts that sit behind their organisation's filtering on top of the provider's.
So two users on the same provider can be sent the same OTP, and one finds it in the inbox while the other finds it in junk. From my side, both of those were successful deliveries.
That is worth knowing for anyone debugging this. When a sending service reports an email as delivered, it means the receiving server accepted it. Where it went after that, inbox or spam, is not something the sender is told.
What I changed
Once I understood all of that, the change I made was a small one. I added a line to the OTP page:
Don't see the email? Please check your junk or spam folder.
I cannot control how each mail provider filters email, and I cannot see the rules a user has set up for themselves. What I can control is what the user is told while they wait for the code.
That line does not stop an OTP from going to spam. It stops a user from deciding the email never arrived, when it is actually one folder away.
What I took away
- SPF, DKIM and DMARC are the entry requirement, not the delivery guarantee. Without them, mail is likely to be rejected or filtered. With them, it gets judged on everything else.
- Each mail provider makes its own decision, based largely on reputation that builds up over time rather than on anything in DNS.
- Each user can override that decision, and the sender never sees it happen.
- Send HTML with a matching plain-text version. The HTML is how the email looks. The text version is for every mail app, reader and filter that would rather have it.
- "Delivered" means accepted, not inboxed.
- Tell users where to look. When deliverability is partly out of your hands, the interface is the part you still control.
Authentication tells a mail provider who sent the email. It does not tell the provider to trust it.