ACME renewals intermittently fail with "404 :: Certificate not found" — hardcoded `--preferred-chain=ISRG Root X1` no longer matches any chain LE offe

[webbasica]

• DA version: 1.709
• lego: 5.4.0 (also reproduced on 5.3.1)
• Challenge: HTTP-01
• OS: AlmaLinux 8.10, OpenLiteSpeed

---

Since Let's Encrypt began issuing from the new YE/YR intermediates, a portion of my renewals fail after validation has already succeeded. Roughly 3 of 20 domains fail per nightly run, and which ones fail changes each night.

Example (domain genericised):

INFO  Trying renewal. cert-name=example.com time-remaining=84d8h15m26s
INFO  Authorization is already valid; skipping the challenge. domain=example.com
INFO  Validations succeeded; requesting certificates. domains="example.com, www.example.com"
WARN  retry: acme: error: 404 :: urn:ietf:params:acme:error:malformed :: Certificate not found
ERROR Error error="renew certificate: could not obtain the certificate for \"example.com\":
      ... POST https://acme-v02.api.letsencrypt.org/acme/cert/<id>/1
      giving up after 1 attempt(s): acme: error: 404 :: urn:ietf:params:acme:error:malformed :: Certificate not found"
exit status 1

The failing request is /acme/cert/<id>/1 — the alternate chain. lego only asks for alternates when the default chain doesn't satisfy --preferred-chain.

Cause

DA invokes lego with --preferred-chain=ISRG Root X1. Under LE's new hierarchy the issued bundle is:

leaf -> YE2 -> ISRG Root YE -> ISRG Root X2 -> ISRG Root X1

The topmost certificate in that bundle is ISRG Root X2. X1 appears only as X2's issuer field, never as a certificate. So lego's preferred-chain matcher finds no chain whose root cert is literally X1, requests alternate /1, and LE answers 404 — inconsistently, which is why it's intermittent and why it varies domain to domain within a single run.

Removing the flag fixes it, and does not change the chain

Re-running DA's exact command with only --preferred-chain removed succeeds immediately:

INFO  Server responded with a certificate. domains="example.com, www.example.com"

The resulting bundle is identical — same three intermediates, same cross-signs, Verify return code: 0 (ok) against the system trust store. The flag is not protecting the cross-signed path to X1; that path is already what LE serves by default.

Ruled out

• lego version — the identical 404 occurs on both 5.3.1 and 5.4.0.
• DNS-01 propagation issues (as in the recent DNS-challenge thread) — this is HTTP-01, and it fails after Validations succeeded.
• Challenge reachability / CAA / firewall — validation completes cleanly; only the chain fetch fails.

Why there's no user-side workaround

The older advice to copy scripts/letsencrypt.sh into scripts/custom/ and edit the lego command no longer applies — on 1.709 that script contains no --preferred-chain and no lego issuance call; it's now a config helper and issuance lives in the binary. There is also no preferred_chain key in directadmin config. So the flag can't be changed without patching the binary or wrapping /usr/local/bin/lego.

Request

Could --preferred-chain be made configurable (e.g. a letsencrypt_preferred_chain conf value, empty = don't pass the flag), or dropped now that the default chain already terminates at ISRG Root X1 via cross-signs?

This will affect more installs as LE completes the YE/YR rollout. acme_disable_after_failures=10 means repeatedly-unlucky domains will eventually have ACME switched off entirely.

 

[fln]

Thanks for the report @webbasica. So far we have not saw similar errors on our test servers.

Historically the --preferred-chain=ISRG Root X1" parameter did not cause any problems, even if certificate is issued from ACME providers that does not support it. Lego accepts the certificate even if it does not have the desired chain cert.

Do you have a way to reproduce this problem? Was the report generate with the help of AI tools?

 

[webbasica]

Yes, I have Claude connected to my servers, helps me with the throubleshooting.

I tried reproducing the error, but that only disproved my own theory, so please disregard the preferred-chain diagnosis.

Running DA's exact lego command by hand, production fails ~1-2 in 3 with 404 :: Certificate not found on the certificate-retrieval POST after Validations succeeded, and it fails identically with --preferred-chain removed. LE staging never reproduces it in 8 runs. So this looks like an intermittent Let's Encrypt production issue where the cert URL 404s just after finalize, and lego gives up after one attempt.

The one thing that may still be actionable on DA's side: a failed renewal isn't retried within the run, so a transient CA-side 404 costs the domain a full cycle and counts toward acme_disable_after_failures. A short retry on 404/5xx after finalize would absorb it.