LetsEncrypt Suddenly Stopped Working

T

TheCableGuy96

Verified User
Joined
Apr 29, 2010
Messages
48
Hi,

I've had my server running for a few years now and keep it updated regularly, it's just a personal web server run from home but has it's own public IP for the web server and another 2 IPs for each DNS server nameserver (both allocated to this server ns1 and ns2).

Everything has been working great until yesterday when I got a notification that it couldn't renew the domains.

I've tried manually and get this error:
=================================================================
2024/10/31 12:31:41 [INFO] [exampledomain.com] acme: Obtaining SAN certificate
2024/10/31 12:31:42 [INFO] [exampledomain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/42368281
2024/10/31 12:31:42 [INFO] [exampledomain.com] acme: Could not find solver for: tls-alpn-01
2024/10/31 12:31:42 [INFO] [exampledomain.com] acme: use http-01 solver
2024/10/31 12:31:42 [INFO] [exampledomain.com] acme: Trying to solve HTTP-01
2024/10/31 12:31:47 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/42368281
2024/10/31 12:31:48 Could not obtain certificates:
error: one or more domains had a problem:
[exampledomain.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: During secondary validation: 2606:xxxx:xxxx::ac43:b49c: Invalid response from https://exampledomain.com/.well-known/acme-challenge/jM_hdpzXPK5ORmtQQ5saY_IBj3uxqCDPfRQRiS: 403
Failed to issue new certificate
=================================================================

I've tried ensuring all system and package updates are installed and rebooted
Tried with firewall off
Ensured Cloudflare "page rule" still exists for the domain:
*exampledomain.com/.well-known/acme-challenge/*
Cache Level: Bypass

I've been Googling for about 3 hours and just cannot get to the bottom of it. Can anyone offer any help please? Nothing has changed at my end so I just cannot understand why it's stopped working randonly.

Thank you.
 
Last edited:
Also be sure if you use more nameserver ip's (and you do as you use cloudflare), the delegation is done on time, so the domain is reachable on every ip, else LE can still fail.
 
Hey, thanks for the reply.

Richard G said:
No access to the file it wants to reach.

You might want to try this and the commands in there to narrow down the issue.

Troubleshooting Let's Encrypt Errors | Directadmin Docs

DirectAdmin Knowledge Base
docs.directadmin.com
Click to expand...

I've looked at this already and went down as far as the test file which was accessible. Following that it was specific errors and fixes for those errors and non were applicable so I didn't see any point in trying those. Unless I'm missing something?

Richard G said:
Also be sure if you use more nameserver ip's (and you do as you use cloudflare), the delegation is done on time, so the domain is reachable on every ip, else LE can still fail.
Click to expand...

Are you referring to /etc/resolv.conf? I already have 2 different providers and it's never been an issue before:
nameserver 8.8.8.8
nameserver 1.1.1.1

I'd be very surprised if it was anything to do with that as the records have been the same for a very long time.

I wonder if it's something to do with this new LEGO system. I don't understand it yet as I've only heard about it today when researching the issue but it appears to be something to do with DNS and DirectAdmin knowledge base says something about it being implemented very recently (last month) so would make sense with the timing and error?

Thanks.
 
TheCableGuy96 said:
Are you referring to /etc/resolv.conf?
Click to expand...
No. I was referring to nameservers you have setup for your domain.
For example ns1.yourdomain.com -> ip. But I don't know how that works with cloudflare as it seems LE is looking at other ip's then the nameservers.

LE tries these for your domain.
104.21.xxx.xxx
1"172.67.xxxx.xxxx"
2"2606:4700:xxxxxxxxx49c"
3"2606:4700:xxxxxxxxx381"

Then it decides to use the 2nd one, the ipv6 and then gets a forbidden notice.

TheCableGuy96 said:
And I was able to access it publicably from https://www.mydomain.com/.well-known/acme-challenge/test.txt
Click to expand...
Seems you removed it again, I got a "not found". :)

However, you have to be sure that LE can access the file from all 4 ip's mentioned here. And the log says "unauthorized" on the secondary validation which it's trying via that ipv6.
So for some reason, the file is not reachable via the ipv6, which might be the secondary resolver. But DNS records for LE must be present at all resolvers or it will fail.

I hope that makes it a bit more clear in which direction you have to look.
 
Yes I did remove it but if you check the domain I supplied I changed it to "mydomain.com" for privacy so it's not my domain.

So I'm getting somewhere with this but still not there yet....

It would appear it's something to do with this "LEGO" system that is DNS related. I'm not clued up on it yet but something has changed in DA and it now requires you to specify the DNS system to do the lookup for your when issuing certs. Luckily there's 2 ways of doing this:
1) Though a file for each domain in: /usr/local/directadmin/data/users/youruser/domains/mydomain.com.dnsprovider You will need one for each domain.
2) Or if you switch to the "Enhanced" skin rather than "Evolution" (it doesn't support it in Evolution) under the "SSL Certificates" there's an option to select from a ton of DNS providers it can use. I entered my Cloudflare details and created the API keys on cloudflare. Also, you can do this under Admin account for the hostname for emails and other system services but this only works in "Evolution" skin rather than "Enhanced" skin this time (strange I know, one skin for user and another skin for the admin).

Now when I run it after entering my Cloudflare details it works.

But the problem is even though it's working it's not being applied. I restarted directadmin and apache and have used a different browser but still when I check the cert in use it's showing the old one rather than the new one. I've even restarted the server.

Why the old lookup system has suddenly stopped working I don't know but I can only assume DA have swapped something as their knowledge base says it's now in DA to specificy DNS providers from last month. So it's either a bug or the old lookup system is unsupported now.

So I need to work out still why the certs are not being utilised after they are being renewed.
 
TheCableGuy96 said:
I changed it to "mydomain.com" for privacy so it's not my domain.
Click to expand...
I know mydomain.com is not your domain, but I wouldn't know the ip's either if I didn't know your real domain name. :)

Seems to me renewing your domain's certificate already worked 3 days ago:
28-10-2024 until 12-12-2024.

I don't know why it's not renewed locally. Indeed the certificate requested in september, working until december 3rd is still in place.

Odd thing is... when looking at crt.sh the older certificate, so the one working until the 3rd, is on top. Normally newest are on top but I also see certificates made october 1st valid until december 30th.
You can check yourself on crt.sh site. But I don't see one from today. However, that's possible, sometimes it takes a bit of time to display.

P.s. can't help you further with this, I don't use cloudflare. Maybe @zEitEr has a clue.
 
Richard G said:
I know mydomain.com is not your domain, but I wouldn't know the ip's either if I didn't know your real domain name. :)

Seems to me renewing your domain's certificate already worked 3 days ago:
28-10-2024 until 12-12-2024.

I don't know why it's not renewed locally. Indeed the certificate requested in september, working until december 3rd is still in place.

Odd thing is... when looking at crt.sh the older certificate, so the one working until the 3rd, is on top. Normally newest are on top but I also see certificates made october 1st valid until december 30th.
You can check yourself on crt.sh site. But I don't see one from today. However, that's possible, sometimes it takes a bit of time to display.

P.s. can't help you further with this, I don't use cloudflare. Maybe @zEitEr has a clue.
Click to expand...

ahhh I guess you've accessed one of my domains through the error links, I missed that so I've edited the URL to protect my privacy.

I've reviewed the site you suggested and it displays the last ones from three days ago as you said but I didn’t initiate these. I’m unsure why two requests occurred on the same day, though it seems consistent with past behaviour, so it might be normal. The recent ones you can't yet see are just the automated attempts that failed and ones I've been trying since receiving renewal error notifications However, I’m uncertain why these are trying to renew automatically, given that a successful renewal occurred just three days ago according to that website.

Currently, the certificates are processing as I’ve added my Cloudflare details for LEGO to perform lookups. The certificates show a 59-day renewal period in DA which is standard for the day they have been processed, but they’re not being applied to the domains or host.

Thanks for your help so far—hopefully, someone else might have further insights.
 
Hello,

A similar case was investigated by me some time ago, it seems CloudFlare blocks access from LetsEncrypt bots. Thus CloudFlare can not verify a domain and therefore denies a certificate renewal. Here are some possible ways to bypass it:

1. switch to ZeroSSL
2. use self-signed certificate (actually it's OK, since your domain is protected by CloudFlare)
3. check settings at CloudFlare and disable a protection against bots (probably it is available only on paid tariff, not too sure).
 
You must log in or register to reply here.
Back
Top