Issues with ACME DNS challenge

zEitEr

zEitEr

Super Moderator
Joined
Apr 11, 2005
Messages
16,264
Location
www.poralix.com
As of yesterday had a couple of servers to work on with a bunch of domains which I had to address manually (they had worked fine for years with valid Letsencrypt certificates). Here is what I found:

1. New ACME feature does propagate to use nameservers from /etc/resolv.conf. For some reasons DNS resolvers from DC are enough slow to catch challenge records in DNS. And a certificate renewal/issue breaks. So here again we need to use none-caching DNS resolvers such as 1.1.1.1,8.8.8.8.

2. Stealth nameservers in a DNS zone break a DNS validation. So I had to clean every domain with wrong nameservers.

Even fixed the both, still some domains fail for unknown reasons. So I had to disable wildcard- certificate mode, which is enabled by default.

Too much pain when users/admins are on vacations alongside with multiple vulnerabilities in WP and kernels.

p.s. Not complaining, but trying to master the new feature. Funny)))
 
@zEitEr, in the last DA release we are adding a special permanent DNS cache busting record:
Code: 
dig +short _acme-challenge.example.com in txt
"persistent-record"

This makes sure that even if you are using a caching DNS resolver it will detect new DNS records in 2 minutes.

This was added in DA 1.707 because lego v5 uses DNS much more aggressively and almost always poison the DNS cache with NXDOMAIN answer before DNS challenge can be solved.


DA uses NS record mismatch to detect if DA is actually managing the domain or not. If system resolver says domain.com uses ns1.example.com and ns2.example.com, but local bind says domain.com uses ns1.server.com and ns2.server.com, this is a flag for DA to assume local bind does not control this domain and to not try DNS challenge with local bind (it would still use DNS challenge it if custom DNS provider is configured).

If DA do actually control the domain DNS zone, then syncing NS records will unlock the automatic ACME DNS challenge. If DA does not actually control the domain DNS zone, then syncing NS records will break ACME DNS challenge.
 
fln said:
@zEitEr, in the last DA release we are adding a special permanent DNS cache busting record:
Click to expand...


Why would DA persist on trying DNS for a domain with wrong nameservers?

Failed with "no such host" error:

Code: 
Failed to issue a TLS certificate for some-hosting-here.nl, *.some-hosting-here.nl DNS names.
--- Command output
2026-08-04T14:08:18.478800702+02:00 INFO  Obtaining SAN certificate. domains="some-hosting-here.nl, *.some-hosting-here.nl"
2026-08-04T14:08:19.305825877+02:00 INFO  Authorization is already valid; skipping the challenge. domain=some-hosting-here.nl
2026-08-04T14:08:19.305954629+02:00 INFO  Use solver. domain=*.some-hosting-here.nl type=dns-01
2026-08-04T14:08:19.306021498+02:00 INFO  dns01: preparing to solve the challenge. domain=*.some-hosting-here.nl
2026-08-04T14:08:20.908828922+02:00 INFO  2026/08/04 14:08:20  info executing task            task=action=dns&do=add&domain=some-hosting-here.nl&name=_acme-challenge&ttl=120&type=TXT&value=%22persistent-record%22
2026-08-04T14:08:21.609531113+02:00 INFO  2026/08/04 14:08:21  info finished task             duration=700.657856ms task=action=dns&do=add&domain=some-hosting-here.nl&name=_acme-challenge&ttl=120&type=TXT&value=%22persistent-record%22
2026-08-04T14:08:22.595703921+02:00 INFO  2026/08/04 14:08:22  info executing task            task=action=dns&do=add&domain=some-hosting-here.nl&name=_acme-challenge&named_reload=yes&ttl=120&type=TXT&value=%22EYJvDjgwzw0sJ48wpCY1mSVs4ZnHZns7qYkDyMlwTXw%22
2026-08-04T14:08:25.447797058+02:00 INFO  2026/08/04 14:08:25  info finished task             duration=2.852790748s task=action=dns&do=add&domain=some-hosting-here.nl&name=_acme-challenge&named_reload=yes&ttl=120&type=TXT&value=%22EYJvDjgwzw0sJ48wpCY1mSVs4ZnHZns7qYkDyMlwTXw%22
2026-08-04T14:08:25.449561299+02:00 INFO  dns01: trying to solve the challenge. domain=*.some-hosting-here.nl
2026-08-04T14:08:25.451230280+02:00 INFO  dns01: waiting for record propagation timeout=5m0s interval=30s domain=*.some-hosting-here.nl
2026-08-04T14:08:55.492523428+02:00 INFO  dns01: waiting for record propagation. domain=*.some-hosting-here.nl
2026-08-04T14:09:25.535715454+02:00 INFO  dns01: waiting for record propagation. domain=*.some-hosting-here.nl
2026-08-04T14:09:55.567085684+02:00 INFO  dns01: waiting for record propagation. domain=*.some-hosting-here.nl
2026-08-04T14:10:25.584110022+02:00 INFO  dns01: waiting for record propagation. domain=*.some-hosting-here.nl
2026-08-04T14:10:55.629399278+02:00 INFO  dns01: waiting for record propagation. domain=*.some-hosting-here.nl
2026-08-04T14:11:25.661063644+02:00 INFO  dns01: waiting for record propagation. domain=*.some-hosting-here.nl
2026-08-04T14:11:55.695945681+02:00 INFO  dns01: waiting for record propagation. domain=*.some-hosting-here.nl
2026-08-04T14:12:25.730335945+02:00 INFO  dns01: waiting for record propagation. domain=*.some-hosting-here.nl
2026-08-04T14:12:55.772213378+02:00 INFO  dns01: waiting for record propagation. domain=*.some-hosting-here.nl
2026-08-04T14:13:25.788622877+02:00 INFO  dns01: waiting for record propagation. domain=*.some-hosting-here.nl
2026-08-04T14:13:55.814759907+02:00 INFO  dns01: cleaning DNS-01 challenge. domain=*.some-hosting-here.nl
2026-08-04T14:13:57.590725943+02:00 INFO  2026/08/04 14:13:57  info executing task            task=action=dns&do=delete&domain=some-hosting-here.nl&name=_acme-challenge&named_reload=yes&type=TXT&value=%22EYJvDjgwzw0sJ48wpCY1mSVs4ZnHZns7qYkDyMlwTXw%22
2026-08-04T14:13:59.809815720+02:00 INFO  2026/08/04 14:13:59  info finished task             duration=2.219632235s task=action=dns&do=delete&domain=some-hosting-here.nl&name=_acme-challenge&named_reload=yes&type=TXT&value=%22EYJvDjgwzw0sJ48wpCY1mSVs4ZnHZns7qYkDyMlwTXw%22
2026-08-04T14:14:00.218425179+02:00 INFO  Skipping deactivating of valid authorization. url=https://acme-v02.api.letsencrypt.org/acme/authz/815647577/749576892911
2026-08-04T14:14:00.347368192+02:00 INFO  Deactivating authorization. url=https://acme-v02.api.letsencrypt.org/acme/authz/815647577/752723352141
2026-08-04T14:14:00.504742136+02:00 ERROR Error error="obtain certificate: resolver: one or more domains had a problem: [*.some-hosting-here.nl: dns01: time limit exceeded: last error: authoritative nameservers: DNS call error: dial udp: lookup ns2.knothhosting39.nl. on 127.0.0.253:53: no such host [ns=ns2.knothhosting39.nl.:53, question='_acme-challenge.some-hosting-here.nl. IN  TXT']]"
---

another domain failed with stealth nameservers with the error "server misbehaving":

Code: 
Failed to issue a TLS certificate for xb*******er.nl, *.xb*******er.nl DNS names.
--- Command output
2026-08-04T13:51:14.559153720+02:00 INFO  Obtaining SAN certificate. domains="xb*******er.nl, *.xb*******er.nl"
2026-08-04T13:51:15.293548023+02:00 INFO  Authorization is already valid; skipping the challenge. domain=xb*******er.nl
2026-08-04T13:51:15.293574825+02:00 INFO  Use solver. domain=*.xb*******er.nl type=dns-01
2026-08-04T13:51:15.293588594+02:00 INFO  dns01: preparing to solve the challenge. domain=*.xb*******er.nl
2026-08-04T13:51:17.410227275+02:00 INFO  2026/08/04 13:51:17  info executing task            task=action=dns&do=add&domain=xb*******er.nl&name=_acme-challenge&ttl=120&type=TXT&value=%22persistent-record%22
2026-08-04T13:51:17.984317151+02:00 INFO  2026/08/04 13:51:17  info finished task             duration=574.189447ms task=action=dns&do=add&domain=xb*******er.nl&name=_acme-challenge&ttl=120&type=TXT&value=%22persistent-record%22
2026-08-04T13:51:18.637222345+02:00 INFO  2026/08/04 13:51:18  info executing task            task=action=dns&do=add&domain=xb*******er.nl&name=_acme-challenge&named_reload=yes&ttl=120&type=TXT&value=%22VnXb_TLkPD_0F0UIWu0XsuNbexfWagfNrXWsmuq5shw%22
2026-08-04T13:51:19.891304738+02:00 INFO  2026/08/04 13:51:19  info finished task             duration=1.251878673s task=action=dns&do=add&domain=xb*******er.nl&name=_acme-challenge&named_reload=yes&ttl=120&type=TXT&value=%22VnXb_TLkPD_0F0UIWu0XsuNbexfWagfNrXWsmuq5shw%22
2026-08-04T13:51:19.915411084+02:00 INFO  dns01: trying to solve the challenge. domain=*.xb*******er.nl
2026-08-04T13:51:19.916063093+02:00 INFO  dns01: waiting for record propagation timeout=5m0s interval=30s domain=*.xb*******er.nl
2026-08-04T13:51:50.235786171+02:00 INFO  dns01: waiting for record propagation. domain=*.xb*******er.nl
2026-08-04T13:52:20.429373473+02:00 INFO  dns01: waiting for record propagation. domain=*.xb*******er.nl
2026-08-04T13:52:50.633755285+02:00 INFO  dns01: waiting for record propagation. domain=*.xb*******er.nl
2026-08-04T13:53:20.994157162+02:00 INFO  dns01: waiting for record propagation. domain=*.xb*******er.nl
2026-08-04T13:53:51.326496917+02:00 INFO  dns01: waiting for record propagation. domain=*.xb*******er.nl
2026-08-04T13:54:21.554639586+02:00 INFO  dns01: waiting for record propagation. domain=*.xb*******er.nl
2026-08-04T13:54:51.811439149+02:00 INFO  dns01: waiting for record propagation. domain=*.xb*******er.nl
2026-08-04T13:55:22.081537465+02:00 INFO  dns01: waiting for record propagation. domain=*.xb*******er.nl
2026-08-04T13:55:52.212475326+02:00 INFO  dns01: waiting for record propagation. domain=*.xb*******er.nl
2026-08-04T13:56:22.488250205+02:00 INFO  dns01: waiting for record propagation. domain=*.xb*******er.nl
2026-08-04T13:56:52.518676468+02:00 INFO  dns01: cleaning DNS-01 challenge. domain=*.xb*******er.nl
2026-08-04T13:56:54.126282552+02:00 INFO  2026/08/04 13:56:54  info executing task            task=action=dns&do=delete&domain=xb*******er.nl&name=_acme-challenge&named_reload=yes&type=TXT&value=%22VnXb_TLkPD_0F0UIWu0XsuNbexfWagfNrXWsmuq5shw%22
2026-08-04T13:56:56.199307022+02:00 INFO  2026/08/04 13:56:56  info finished task             duration=2.072896434s task=action=dns&do=delete&domain=xb*******er.nl&name=_acme-challenge&named_reload=yes&type=TXT&value=%22VnXb_TLkPD_0F0UIWu0XsuNbexfWagfNrXWsmuq5shw%22
2026-08-04T13:56:56.649055964+02:00 INFO  Skipping deactivating of valid authorization. url=https://acme-v02.api.letsencrypt.org/acme/authz/815647577/742911946965
2026-08-04T13:56:56.801267327+02:00 INFO  Deactivating authorization. url=https://acme-v02.api.letsencrypt.org/acme/authz/815647577/752715420361
2026-08-04T13:56:56.984392441+02:00 ERROR Error error="obtain certificate: resolver: one or more domains had a problem: [*.xb*******er.nl: dns01: time limit exceeded: last error: authoritative nameservers: DNS call error: dial udp: lookup ns2.none-existing.nl. on 127.0.0.253:53: server misbehaving [ns=ns2.none-existing.nl.:53, question='_acme-challenge.xb*******er.nl. IN  TXT']]"
---
 
All the checks DA perform to select DNS vs HTTP challenge happens before lego is command is executed. To get more insight why DNS challenge was performed try running the cert taskq action in debug mode. Examples:
Code: 
# da taskq --debug 1 --run 'action=rewrite&value=letsencrypt&domain=example1.com'
2026/08/05 20:18:29  info executing task            task=action=rewrite&domain=example1.com&value=letsencrypt
2026/08/05 20:18:29 debug flushing unbound dns zone acme-mode=domain domain=example1.com hostname=example1.com output=ok removed 5 rrsets, 3 messages and 1 key entries
 user=john zone=example1.com
2026/08/05 20:18:29 debug domain should be able to solve DNS challenge acme-mode=domain domain=example1.com hostname=example1.com reason=ns_record_match user=john
2026/08/05 20:18:29 debug DNS name skipped because it is already covered by a valid certificate acme-mode=domain dnsName=subdomain.example1.com domain=example1.com user=john
2026/08/05 20:18:29 debug DNS name skipped because it is present in the domain configuration acme_skip_dns_names list acme-mode=domain dnsName=example1.com domain=example1.com user=john
2026/08/05 20:18:29  info finished task             duration=284.652518ms task=action=rewrite&domain=example1.com&value=letsencrypt

Check the line domain should be able to solve DNS challenge and the field reason.
The output you are checking are coming from the lego CLI tool. Lego v5 heavily refactored how it checks if DNS challenge is ready. At first we disabled some of the new checks, but with the cache busting entries default lego checks seems to be fine. If you are using lego v5 and it is blocking the DNS challenge you can try passing the --dns-persist.propagation.disable-ans or --dns-persist.propagation.disable-rns options (or both).
 
You feedback is really appreciated. With the DA 1.707 support for lego v5 adds more permutations the the already complicated situation.
 
You must log in or register to reply here.
Back
Top