Let's Encrypt Wildcard Requests Failing

[simplificare]

I've been seeing Let's Encrypt, automatic SSL renewals fail across my servers over the past two months and I'm not sure why... I've been racking my brain at the issue and need another set of eyes.

It all starts with an Auto SSL renewal failing -- Getting a notification in the message system saying the renewal failed, upon investigation it looks like the DNS Propagation is the failure point. -- When I try to renew manually, I get the same error.

Found wildcard domain name and http challenge type, switching to dns-01 validation.
2024/02/29 00:11:28 [INFO] [*.heartstrong.life, heartstrong.life] acme: Obtaining SAN certificate
2024/02/29 00:11:29 [INFO] [*.heartstrong.life] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/320713853717
2024/02/29 00:11:29 [INFO] [heartstrong.life] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/320713853727
2024/02/29 00:11:29 [INFO] [*.heartstrong.life] acme: use dns-01 solver
2024/02/29 00:11:29 [INFO] [heartstrong.life] acme: Could not find solver for: tls-alpn-01
2024/02/29 00:11:29 [INFO] [heartstrong.life] acme: Could not find solver for: http-01
2024/02/29 00:11:29 [INFO] [heartstrong.life] acme: use dns-01 solver
2024/02/29 00:11:29 [INFO] [*.heartstrong.life] acme: Preparing to solve DNS-01
2024/02/29 00:11:31 2024/02/29 00:11:29  info executing task            task=action=dns&do=delete&domain=heartstrong.life&name=_acme-challenge&type=TXT
2024/02/29 00:11:30  info executing task            task=action=dns&do=add&domain=heartstrong.life&name=_acme-challenge&named_reload=yes&ttl=5&type=TXT&value=%22toiETefFq5iH6VuzpJBnqgvBhfXtk6P7PmuQu6bPEKI%22

2024/02/29 00:11:31 [INFO] [*.heartstrong.life] acme: Trying to solve DNS-01
2024/02/29 00:11:31 [INFO] [*.heartstrong.life] acme: Checking DNS record propagation using [8.8.8.8:53]
2024/02/29 00:12:01 [INFO] Wait for propagation [timeout: 5m0s, interval: 30s]
2024/02/29 00:12:01 [INFO] [*.heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:12:31 [INFO] [*.heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:13:02 [INFO] [*.heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:13:32 [INFO] [*.heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:14:02 [INFO] [*.heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:14:32 [INFO] [*.heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:15:02 [INFO] [*.heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:15:32 [INFO] [*.heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:16:02 [INFO] [*.heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:16:32 [INFO] [*.heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:17:02 [INFO] [*.heartstrong.life] acme: Cleaning DNS-01 challenge
2024/02/29 00:17:03 2024/02/29 00:17:02  info executing task            task=action=dns&do=delete&domain=heartstrong.life&name=_acme-challenge&type=TXT

2024/02/29 00:17:03 [INFO] [heartstrong.life] acme: Preparing to solve DNS-01
2024/02/29 00:17:05 2024/02/29 00:17:03  info executing task            task=action=dns&do=delete&domain=heartstrong.life&name=_acme-challenge&type=TXT
2024/02/29 00:17:04  info executing task            task=action=dns&do=add&domain=heartstrong.life&name=_acme-challenge&named_reload=yes&ttl=5&type=TXT&value=%22tG7lgcYoyHmv6PdGoas-I5ChZK_fAvoi7Q378XZTGV4%22

2024/02/29 00:17:05 [INFO] [heartstrong.life] acme: Trying to solve DNS-01
2024/02/29 00:17:05 [INFO] [heartstrong.life] acme: Checking DNS record propagation using [8.8.8.8:53]
2024/02/29 00:17:35 [INFO] Wait for propagation [timeout: 5m0s, interval: 30s]
2024/02/29 00:17:35 [INFO] [heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:18:05 [INFO] [heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:18:36 [INFO] [heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:19:06 [INFO] [heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:19:36 [INFO] [heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:20:06 [INFO] [heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:20:36 [INFO] [heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:21:06 [INFO] [heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:21:36 [INFO] [heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:22:06 [INFO] [heartstrong.life] acme: Waiting for DNS record propagation.
2024/02/29 00:22:36 [INFO] [heartstrong.life] acme: Cleaning DNS-01 challenge
2024/02/29 00:22:37 2024/02/29 00:22:36  info executing task            task=action=dns&do=delete&domain=heartstrong.life&name=_acme-challenge&type=TXT

2024/02/29 00:22:37 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/320713853717
2024/02/29 00:22:37 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/320713853727
2024/02/29 00:22:37 Could not obtain certificates:
    error: one or more domains had a problem:
[*.heartstrong.life] time limit exceeded: last error: NS ns2.simplificare-dns.com. returned NXDOMAIN for _acme-challenge.heartstrong.life.
[heartstrong.life] time limit exceeded: last error: NS ns2.simplificare-dns.com. returned NXDOMAIN for _acme-challenge.heartstrong.life.
Failed to issue new certificate

I'm able to test while a renewal is in progress using dig or nslookup (both at 8.8.8.8 and at the local NS) that the acme record is published. I can also see it using whatsmydns.com, however for some reason Let's Encrypt isn't seeing it.

Does anyone see anything else that I might be missing?

Thanks!

 

[Richard G]

Does anyone see anything else that I might be missing?

Not here, it looks all OK.

I did discovered that your ns2 does not contain a GLUE record as does your ns1. Those are external nameservers if I've seen correctly.
Do you maintain those too? Because in that case you could fix that part.

However, since this normally only requires an extra A record lookup, can't imagine that LE trips about that, but one never knows.

I'm able to test while a renewal is in progress using dig or nslookup (both at 8.8.8.8 and at the local NS)

Can you try again and then check your external NS which you are using. Start with checking the ns2.sixxx-xxx.com if the acme_challenge record get's put in there, and after that the ns1.

 

[simplificare]

Do you maintain those too?

I do maintain them. Both ns1 & ns2 are two different DirectAdmin boxes. I run about seven or so servers. Two of which are "main" name servers for all the accounts that I host, all the others servers report theirs zones to the two servers using DA Multi-Server setup.

I'll continue testing a bit later tonight.

Thanks for your reply!

 

[simplificare]

Can you try again and then check your external NS which you are using. Start with checking the ns2.sixxx-xxx.com if the acme_challenge record get's put in there, and after that the ns1.

The acme challenge does indeed get published to ns1, ns2 and I also see it at Google and Cloudflare DNS when I dig them as well. I beleive I found an issue with the local 127.0.0.1 server and have since resolved that, which seems to have resolved the SSL request.

I've clearly configured ns1 to report glue correctly but am struggling to get ns2 to report glue. Any tips you send along would be helpful!

Thanks again!

 

[Richard G]

I beleive I found an issue with the local 127.0.0.1 server and have since resolved that,

Can you tell us what it was and how you resolved it? Would be very helpfull maybe later on with others having the same issue. Next to the fact that I'm also very curious to the cause.

As for the report glue on NS2, that seems almost certain to be a false positive from intodns. Because if I check intodns with the NS domain name, it does not give such error. And that is more important.
I also doublechecked with some other tools and commands and these also all looked good.

 

[simplificare]

It looks like you were right about it not appearing on NS2 properly.

For some reason I had an entry for "ns2.simplificare-dns.com" on the same server that had the account that had failing requests. Despite ns2 running on a different server, setup correctly in multi-server setup, it wasn't publishing the acme-challenge to the actual NS2 server. When I would test the acme-challenge on ns2, it was appearing because it technically was published there... but not on the actual NS. -- Its confusing, but I only noticed it when I tried to publish a record for another domain and it didn't show up on both NS's.

I know I've fixed it now and thought maybe that was causing the GLUE issue too, but agree with your assessment that it might be an intoDNS issue as I don't see any records missing.

Thanks again!

 

[Richard G]

Great! Thank you for your feedback on this.

I hope with you having fixed this now, LE issues will belong definately to the past.