Let's Encrypt not renewing: i/o timeout certificate generation failed

ASap

Verified User
Joined
May 3, 2013
Messages
100
Location
the land of the long white cloud
Hi everyone,

We're having an issue with Let's Encrypt where it doesn't renew domains as expected:


Cannot Execute Your Request​


Details

1
2020/10/15 09:07:20 No key found for account [email protected]. Generating a 4096 key.
2020/10/15 09:07:21 Saved key to /usr/local/directadmin/data/.lego/accounts/acme-v02.api.letsencrypt.org/[email protected]/keys/[email protected]
2020/10/15 09:07:41 Could not create client: get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get "https://acme-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-v02.api.letsencrypt.org on 10.0.0.11:53: read udp 10.0.17.2:56365->10.0.0.11:53: i/o timeout
Certificate generation failed.​


Hints:
10.0.17.2 is our server LAN IP;
10.0.0.11 is our DNS server.

Any clue what went wrong?

Checked custom build and found no update available.
 
Can the lan server & the dns server communicate without restrictions (whitelisted in each other’s firewalls)? The following seems to indicate that either the lan server couldn’t retrieve the result of the dns lookup from the dns server or that the dns server couldn’t resolve LE:
Code:
dial tcp: lookup acme-v02.api.letsencrypt.org on 10.0.0.11:53: read udp 10.0.17.2:56365->10.0.0.11:53: i/o timeout
 
Ahhh, is there a caa record for the domain? Enter the domain at letsdebug.net & see what it says.
 
From within both servers, can you try this:

Bash:
curl -Iv acme-v02.api.letsencrypt.org
and
Bash:
dig a +short acme-v02.api.letsencrypt.org

Also, what is the result of this dig using your domain?

Bash:
dig caa +short domain.tld
 
From within both servers, can you try this:

Bash:
curl -Iv acme-v02.api.letsencrypt.org
and
Bash:
dig a +short acme-v02.api.letsencrypt.org

Also, what is the result of this dig using your domain?

Bash:
dig caa +short domain.tld
Code:
user@server:~$ curl -Iv acme-v02.api.letsencrypt.org
*   Trying 172.65.32.248:80...
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 80 (#0)
> HEAD / HTTP/1.1
> Host: acme-v02.api.letsencrypt.org
> User-Agent: curl/7.72.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Server: nginx
Server: nginx
< Date: Wed, 21 Oct 2020 07:23:30 GMT
Date: Wed, 21 Oct 2020 07:23:30 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 162
Content-Length: 162
< Connection: keep-alive
Connection: keep-alive
< Location: https://acme-v02.api.letsencrypt.org/
Location: https://acme-v02.api.letsencrypt.org/

<
* Connection #0 to host acme-v02.api.letsencrypt.org left intact
user@server:~$ dig a +short acme-v02.api.letsencrypt.org
prod.api.letsencrypt.org.
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
172.65.32.248
user@server:~$ dig caa +short the.host.name
1.2.3.4
user@server:~$

everything looks normal to me (correct me if I'm wrong)
 
Code:
user@server:~$ curl -Iv acme-v02.api.letsencrypt.org
*   Trying 172.65.32.248:80...
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 80 (#0)
> HEAD / HTTP/1.1
> Host: acme-v02.api.letsencrypt.org
> User-Agent: curl/7.72.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Server: nginx
Server: nginx
< Date: Wed, 21 Oct 2020 07:23:30 GMT
Date: Wed, 21 Oct 2020 07:23:30 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 162
Content-Length: 162
< Connection: keep-alive
Connection: keep-alive
< Location: https://acme-v02.api.letsencrypt.org/
Location: https://acme-v02.api.letsencrypt.org/

<
* Connection #0 to host acme-v02.api.letsencrypt.org left intact
user@server:~$ dig a +short acme-v02.api.letsencrypt.org
prod.api.letsencrypt.org.
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
172.65.32.248
user@server:~$ dig caa +short the.host.name
1.2.3.4
user@server:~$

everything looks normal to me (correct me if I'm wrong)
Your CAA record for your domain returns an IP? I believe it should return the name of the Certificate Authority allowed to issue SSLs for your domain (or be empty/not exist as to allow all CAs to issue SSLs for your domain):

Example per https://geekflare.com/dns-caa-record/


Code:
~$ dig caa +short gf.dev
0 issue "digicert.com; cansignhttpexchanges=yes"
0 issuewild "digicert.com; cansignhttpexchanges=yes"
0 issue "comodoca.com"
0 issue "digicert.com"
0 issue "letsencrypt.org"
0 issuewild "comodoca.com"
0 issuewild "digicert.com"
0 issuewild "letsencrypt.org"
~$
 
Your CAA record for your domain returns an IP? I believe it should return the name of the Certificate Authority allowed to issue SSLs for your domain (or be empty/not exist as to allow all CAs to issue SSLs for your domain):

Example per https://geekflare.com/dns-caa-record/


Code:
~$ dig caa +short gf.dev
0 issue "digicert.com; cansignhttpexchanges=yes"
0 issuewild "digicert.com; cansignhttpexchanges=yes"
0 issue "comodoca.com"
0 issue "digicert.com"
0 issue "letsencrypt.org"
0 issuewild "comodoca.com"
0 issuewild "digicert.com"
0 issuewild "letsencrypt.org"
~$
yes, it returned an IP address.
 
Please execute the script in terminal like:
Code:
bash -x /usr/local/directadmin/scripts/letsencrypt.sh request domain.com

That way you'll see what's the query before the output of the error.
 
Please execute the script in terminal like:
Code:
bash -x /usr/local/directadmin/scripts/letsencrypt.sh request domain.com

That way you'll see what's the query before the output of the error.
I don't use 8.8.8.8 resolver but for some reason I found 8.8.8.8 on the output down below:
Code:
root@server:~# bash -x /usr/local/directadmin/scripts/letsencrypt.sh request the.host.name
++ /usr/bin/id -u
+ MYUID=0
+ '[' 0 '!=' 0 ']'
+ export EXEC_PROPAGATION_TIMEOUT=300
+ EXEC_PROPAGATION_TIMEOUT=300
+ export EXEC_POLLING_INTERVAL=5
+ EXEC_POLLING_INTERVAL=5
+ LEGO=/usr/local/bin/lego
+ DNS_SERVER=8.8.8.8
+ DNS6_SERVER=2001:4860:4860::8888
+ NEW_IP=1.1.1.1
+ NEW6_IP=2606:4700:4700::1111
+ DA_IPV6=false
+ TASK_QUEUE=/usr/local/directadmin/data/task.queue.cb
+ LEGO_DATA_PATH=/usr/local/directadmin/data/.lego
+ '[' 2 -lt 2 ']'
+ '[' 2 -lt 3 ']'
+ KEY_SIZE=ec256
+ ECC_USED=true
+ ECC=secp384r1
+ KEY_SIZE=
+ '[' '' = secp384r1 ']'
+ '[' '' = prime256v1 ']'
+ '[' '' = 4096 ']'
+ '[' '' = 2048 ']'
+ '[' '' = 8192 ']'
+ ECC=prime256v1
+ KEY_SIZE=ec256
+ ECC_USED=true
+ DA_BIN=/usr/local/directadmin/directadmin
+ '[' '!' -s /usr/local/directadmin/directadmin ']'
+ '[' request = present ']'
+ '[' request = cleanup ']'
+ /usr/local/directadmin/directadmin c
+ grep -m1 -q '^ipv6=1$'
+ CURL=/usr/local/bin/curl
+ '[' '!' -x /usr/local/bin/curl ']'
+ DIG=/usr/bin/dig
+ '[' '!' -x /usr/bin/dig ']'
+ '[' '' = yes ']'
+ API_URI=acme-v02.api.letsencrypt.org
+ API=https://acme-v02.api.letsencrypt.org
+ CHALLENGETYPE=http
+ GENERAL_TIMEOUT=40
+ CURL_OPTIONS='--connect-timeout 40 -k --silent'
++ uname
+ OS=Linux
+ OPENSSL=/usr/bin/openssl
++ date +%s
+ TIMESTAMP=1603315621
++ /usr/local/directadmin/directadmin c
++ grep '^letsencrypt='
++ cut -d= -f2
+ LETSENCRYPT_OPTION=1
++ /usr/local/directadmin/directadmin c
++ grep '^secure_access_group='
++ cut -d= -f2
+ ACCESS_GROUP_OPTION=access
+ FILE_CHOWN=diradmin:mail
+ FILE_CHMOD=640
+ '[' access '!=' '' ']'
+ FILE_CHOWN=diradmin:access
+ '[' '!' -x /usr/local/bin/lego ']'
+ DOCUMENT_ROOT=
+ WELLKNOWN_PATH=/var/www/html/.well-known/acme-challenge
+ '[' '!' -z '' ']'
+ APPEND_SERVER='-s https://acme-v02.api.letsencrypt.org/directory'
++ hostname -f
+ SERVER_HOSTNAME=cana.itmate.net
+ '[' -z cana.itmate.net ']'
+ '[' '!' -s /usr/local/directadmin/data/users/admin/user.conf ']'
+ ADMIN_USERCONF=/usr/local/directadmin/data/users/admin/user.conf
+ '[' '!' -z /usr/local/directadmin/data/users/admin/user.conf ']'
+ '[' -s /usr/local/directadmin/data/users/admin/user.conf ']'
++ grep -m1 '^email=' /usr/local/directadmin/data/users/admin/user.conf
++ cut -d= -f2
++ cut -d, -f1
+ [email protected]
+ '[' -z [email protected] ']'
+ DOMAIN=the.host.name
+ '[' '' '!=' yes ']'
+ FOUNDDOMAIN=0
++ echo the.host.name
++ tr , ' '
+ for TDOMAIN in '`echo "${DOMAIN}" | tr '\'','\'' '\'' '\''`'
+ DOMAIN_NAME_FOUND=the.host.name
++ echo the.host.name
++ perl -p0 -e 's#\.#\\.#g'
+ DOMAIN_ESCAPED='the\.host\.name'
+ grep -m1 -q '^the\.host\.name:' /etc/virtual/domainowners
++ grep -m1 '^the\.host\.name:' /etc/virtual/domainowners
++ cut '-d ' -f2
+ USER=revive
+ HOSTNAME=0
+ FOUNDDOMAIN=1
+ break
+ '[' 1 -eq 0 ']'
+ CSR_CF_FILE=
+ DA_USERDIR=/usr/local/directadmin/data/users/revive
+ DA_CONFDIR=/usr/local/directadmin/conf
+ HOSTNAME_DIR=/var/www/html
+ '[' '!' -d /usr/local/directadmin/data/users/revive ']'
+ '[' '!' -d /usr/local/directadmin/conf ']'
+ '[' 0 -eq 0 ']'
+ DNSPROVIDER_FALLBACK=/usr/local/directadmin/data/users/revive/domains/the.host.name.dnsprovider
+ '[' -s /usr/local/directadmin/data/users/revive/domains/the.host.name.dnsprovider ']'
+ KEY=/usr/local/directadmin/data/users/revive/domains/the.host.name.key
+ CERT=/usr/local/directadmin/data/users/revive/domains/the.host.name.cert
+ CACERT=/usr/local/directadmin/data/users/revive/domains/the.host.name.cacert
+ '[' '' '!=' '' ']'
+ /usr/local/directadmin/directadmin c
+ grep -m1 -q '^letsencrypt=2$'
+ DOMAIN_DIR=/var/www/html
+ WELLKNOWN_PATH=/var/www/html/.well-known/acme-challenge
+ '[' -s /usr/local/directadmin/data/users/revive/domains/the.host.name.cert ']'
+ '[' request = renew ']'
+ '[' request = request ']'
+ echo the.host.name
+ grep -m1 -q ,
+ '[' -s '' ']'
+ '[' -s /usr/local/directadmin/data/users/revive/domains/the.host.name.cert ']'
+ /usr/bin/openssl x509 -text -noout -in /usr/local/directadmin/data/users/revive/domains/the.host.name.cert
+ grep -m1 -q 'Subject Alternative Name:'
++ /usr/bin/openssl x509 -text -noout -in /usr/local/directadmin/data/users/revive/domains/the.host.name.cert
++ grep -m1 'Subject Alternative Name:' -A1
++ grep DNS:
++ perl -p0 -e 's|DNS:||g'
++ tr -d ' '
+ DOMAIN=the.host.name
+ '[' '!' -e /var/www/html ']'
+ echo the.host.name
+ grep -m1 -q ,
+ DOMAINS=the.host.name
+ DOMAIN_FLAG='-d the.host.name'
+ FIRST_DOMAIN=the.host.name
+ CHALLENGETYPE=http
+ '[' -s /usr/local/directadmin/data/users/revive/domains/the.host.name.dnsprovider ']'
+ '[' '!' -z '' ']'
+ echo '-d the.host.name'
+ grep -m1 -q '*\.'
++ echo the.host.name
++ perl -p0 -e 's/,/ /g'
++ perl -p0 -e 's/^\*.//g'
+ for domain_name in '`echo ${DOMAIN} | perl -p0 -e "s/,/ /g" | perl -p0 -e "s/^\*.//g"`'
+ caa_check the.host.name
+ CAA_OK=true
++ /usr/bin/dig @8.8.8.8 AAAA the.host.name +short
++ tail -n1
++ grep -v '\.$'
+ IP_TO_RESOLV=
+ '[' 0 -eq 9 ']'
++ echo the.host.name
++ awk -F. '{b=$NF;for(i=NF-1;i>0;i--){b=$i FS b;print b}}'
+ for i in '`echo ${1} | awk -F'\''.'\'' '\''{b=$NF;for(i=NF-1;i>0;i--){b=$i FS b;print b}}'\''`'
+ /usr/bin/dig CAA host.name @8.8.8.8 +short
+ grep -m1 -q -F -- issue
+ /usr/bin/dig CAA host.name @8.8.8.8
+ grep -m1 -q -F -- SERVFAIL
+ CAA_OK=false
+ CAA_CURRENT=SERVFAIL
+ for i in '`echo ${1} | awk -F'\''.'\'' '\''{b=$NF;for(i=NF-1;i>0;i--){b=$i FS b;print b}}'\''`'
+ /usr/bin/dig CAA the.host.name @8.8.8.8 +short
+ grep -m1 -q -F -- issue
+ /usr/bin/dig CAA the.host.name @8.8.8.8
+ grep -m1 -q -F -- SERVFAIL
+ CAA_OK=false
+ CAA_CURRENT=SERVFAIL
+ false
+ echo 'CAA record prevents issuing the certificate: SERVFAIL'
CAA record prevents issuing the certificate: SERVFAIL
+ exit 1
root@server:~#
 
I had a client that had this problem and it was a cloudflare problem. I think there is a setting or something on cloudflare. I don't know. I pointed him in the direction of cloudflare and a few minutes later it was fixed.
 
The primary nameservers used for the Let’s Encrypt script is 8.8.8.8, so that’s probably why you see that. As for your CAA record:
https://developers.cloudflare.com/s...rtificate-authority-authorization-caa-records


If your customer has CAA records set on their domain, they will either need to add the following (or remove CAA entirely):

Code:
example.com. IN CAA 0 issue "digicert.com"
example.com. IN CAA 0 issue "letsencrypt.org"
 
Check:
Code:
dig CAA the.host.name @8.8.8.8

It likely returns a SERVFAIL. It could be related to wrongly setup DNSSEC.
 
The primary nameservers used for the Let’s Encrypt script is 8.8.8.8, so that’s probably why you see that. As for your CAA record:
https://developers.cloudflare.com/s...rtificate-authority-authorization-caa-records


If your customer has CAA records set on their domain, they will either need to add the following (or remove CAA entirely):

Code:
example.com. IN CAA 0 issue "digicert.com"
example.com. IN CAA 0 issue "letsencrypt.org"
Code:
 andy@mm  ~  dig CAA the.host.name @8.8.8.8

; <<>> DiG 9.10.6 <<>> CAA the.host.name @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6637
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;the.host.name.        IN    CAA

;; AUTHORITY SECTION:
host.name.        1799    IN    SOA    eva.ns.cloudflare.com. dns.cloudflare.com. 2035498522 10000 2400 604800 3600

;; Query time: 178 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Oct 23 08:17:09 NZDT 2020
;; MSG SIZE  rcvd: 107

 andy@mm  ~ 
 
Back
Top