Windows 7 chrome let’s encrypt problems

system-admin

Verified User
Joined
Aug 1, 2019
Messages
59
For last 24 hours, I have started receiving complaints from several clients. After analyzing, I found that PCs which are having Windows 7 and latest Google Chrome, they are getting errors "Certificate error". Chrome is fully updated: Version 94.0.4606.61 (Official Build) (64-bit)

Why is this happening? From reading around this wasn’t supposed to happen with Windows 7, and that the problematic OS would be Win XP less than SP3. Plus, why Firefox in Windows 7 does not have this problem?

Is there a solution to this other than moving away from let’s encrypt, which is something I would like to avoid? As you can see in the screenshots, Chrome in Windows 7 still has the old expired ones.

lQdJC.png
 

apogee

Verified User
Joined
Jul 6, 2019
Messages
123
Location
Europe
Had the same also with mail on all our cpanel servers, but not on any DA server yet. Solution was to update the cabundle
 

akadi81

Verified User
Joined
Feb 26, 2015
Messages
39
My solution after too many hours:

Ubuntu: apt install ca-certificates
Centos: yum install ca-certificates

file_get_contents works, wget works, any browser works.
 

Richard G

Verified User
Joined
Jul 6, 2008
Messages
7,664
Location
Maastricht
After analyzing, I found that PCs which are having Windows 7 and latest Google Chrome, they are getting errors "Certificate error". Chrome is fully updated: Version 94.0.4606.61 (Official Build) (64-bit)
People should stop using Windows 7 anyway, it's a longer time unsupported.

Thanks... but it did not help :(
Ofcourse not, because those are Linux solutions, not Windows solutions.
You could try Windows Update, but since W7 won't get updates anymore, there is a chance that this root certificate update won't be provided either.

Try this:
 

system-admin

Verified User
Joined
Aug 1, 2019
Messages
59
People should stop using Windows 7 anyway, it's a longer time unsupported.


Ofcourse not, because those are Linux solutions, not Windows solutions.
You could try Windows Update, but since W7 won't get updates anymore, there is a chance that this root certificate update won't be provided either.

Try this:

Yes, people should stop using Windows 7 but I cannot force them to do so. There are 100s of clients. Practically, not possible. As an host, I can only recommend/suggest them but finally it is them, need to take action.

Thanks for the Microsoft Link... I tried it but it didn't work :(

Thanks anyways for your help. Much appreciated :)
 

ikkeben

Verified User
Joined
May 22, 2014
Messages
1,457
Location
Netherlands Germany
Uh that article is speaking about weeks UHUM yup way behind information date Letsencrypt did tell most off us
As written below in that article also!
Alternatively, administrators can look at alternate trust paths by using the intermediate certificate that Let's Encrypt has set up or following suggested configurations from their May bulletin."

Sorry but it was (should) well known about 15 May 2021

So then there was time enough to handle and take action ?

Me and others did post here in Forum to!

----- Original Message --------
Onderwerp: Let's Encrypt - Change for older browsers/devices (15. Mai 2021,
Van: [email protected]

Hello from the staff at Let's Encrypt.

On September 30, there will be a change in how older browsers and
devices trust Let's Encrypt certificates, resulting in a minor decrease
in compatibility. If you run a typical website, you won't notice a
difference. Devices and browsers running up-to-date software will
continue working fine, and we've taken steps to make sure the vast
majority of older devices will too. If you run a large website, or need
to support less common software (particularly non-browser software),
you'll want to read about the details at:


In either case, no action is required from you. We're letting you know
so you can provide answers to any questions your site visitors may have.

Here is a sample hostname from one of your current Let's Encrypt
certificates: limits.eu

Since 2015 we've served the world with 1.6 billion free certificates,
each one providing security and privacy to people on the Web. It's work
that's 100% funded by charitable donations since we are a nonprofit. If
your company is interested in sponsorship, please email
sponso[email protected]. If you can make a donation, we ask that you
consider supporting our work today: https://letsencrypt.org/donate/
Thank you.
 

fln

Administrator
Staff member
Joined
Aug 30, 2021
Messages
96
hi everyone,

This LE root cert expiration have definitely caused some trouble for old installations. However we have just released an update to DA letsencrypt package to help servers support more old operating systems. Here is the story:

Up till now when LE certificates were used in DA we would serve the following certificate chain for clients connecting to DA or main web server:


Code:
# openssl s_client -connect example.net:2222
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
---
Certificate chain
 0 s:/CN=example.net
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
...

The server would respond with three certificates 0 is the server cert, 1 is the LE intermediate cert and 3 is the LE root CA cert. On September 30th DST Root CA X3 certificate have expired, but this should have not caused problems for all moderately old systems that has ISRG Root X1 CA certificate installed in the system.

For those systems having ISRG Root X1 certificate served from the web server were optional, because those systems already trusts for ISRG Root X1 to be reliable CA cert. However it was important for even older systems that trusted DST Root CA X3 but were not aware about the ISRG Root X1 existence.

After September 30th 2021 those extremely old systems can not connect to servers using LE certificates and even including ISRG Root X1 in the server cert chain can not help them. Because DST Root CA X3 is dead for everyone.

However expiration of the DST Root CA X3 cert revealed another issue. Old software using old openssl versions refuses to connect to DA web server even when they do have DST Root CA X3 trusted as root cert! This is because they see a reference to no longer valid DST Root CA X3 certificate. This issue is not present in newer library version but still affects older (but not very old systems) where openssl is old but the system would otherwise trust ISRG Root X1.

For those cases we have released an update to the `letsencrypt` package. The key change is that we no longer include the ISRG Root X1 certificate in web server responses. The response from server with upgraded `letsencrypt` package would look like:

Code:
# openssl s_client -connect example.net:2222
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = example.net
verify return:1
---
Certificate chain
 0 s:/CN=example.net
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
...

We expect the system to have ISRG Root X1 cert present in the system and only provide server certificate and intermediate certificate R3. This allows older (but not extremely old) systems to connect to DA using LE certificates successfully which would otherwise fail if DST Root CA X3 were present in the response.

To download the update please follow these steps:

Code:
# /usr/local/directadmin/custombuild/build update
# /usr/local/directadmin/custombuild/build letsencrypt
# /usr/local/directadmin/scripts/letsencrypt.sh request your.server.name.example.net

Essentially these steps will:
* Update custombuild installation to know about new package versions
* Download latest letsencrypt package (version 2.0.23)
* Re-issue your main server certificate (and omit ISRG Root X1 in the chain)

TL;DR
Upgrading letsencrypt v2.0.23 package via the custombuild can allow some moderately old systems to connect to DA which currently fails to do so. New systems works fine with or without this upgrade. For extremely old systems there is nothing that can be done (except upgrading them).
 

Richard G

Verified User
Joined
Jul 6, 2008
Messages
7,664
Location
Maastricht
As an host, I can only recommend/suggest them but finally it is them, need to take action.
Correct, but as a host you can also say you can't help that things don't work anymore in unsupported systems. XP users can't expect for things to work either. But I understand where you're coming from. If you have that many people using W7, you still want to provide them with some kind of service.

However, as for my attempt to help, you're welcome. Luckily fln has posted a solution and I presume that you updated you're chrome too now. ;)
 

Richard G

Verified User
Joined
Jul 6, 2008
Messages
7,664
Location
Maastricht
/usr/local/directadmin/scripts/letsencrypt.sh request your.server.name.example.net
Shouldn't that be:
/usr/local/directadmin/scripts/letsencrypt.sh request_single your.server.name.example.net 4096 ??
or even easier:
/usr/local/directadmin/scripts/letsencrypt.sh request_single `hostname` 4096
 
Last edited:

ikkeben

Verified User
Joined
May 22, 2014
Messages
1,457
Location
Netherlands Germany
You can check parts here


domain name then id ( in the list at domainname you see yes no x3... ) and even some advanced also ocsp

and do some read for centos 7 here https://community.letsencrypt.org/t...t-compatibility-after-new-chain/151969?page=2

Custom Python application info when problems
 
Last edited:

system-admin

Verified User
Joined
Aug 1, 2019
Messages
59
hi everyone,

This LE root cert expiration have definitely caused some trouble for old installations. However we have just released an update to DA letsencrypt package to help servers support more old operating systems. Here is the story:

Up till now when LE certificates were used in DA we would serve the following certificate chain for clients connecting to DA or main web server:


Code:
# openssl s_client -connect example.net:2222
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
---
Certificate chain
 0 s:/CN=example.net
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
...

The server would respond with three certificates 0 is the server cert, 1 is the LE intermediate cert and 3 is the LE root CA cert. On September 30th DST Root CA X3 certificate have expired, but this should have not caused problems for all moderately old systems that has ISRG Root X1 CA certificate installed in the system.

For those systems having ISRG Root X1 certificate served from the web server were optional, because those systems already trusts for ISRG Root X1 to be reliable CA cert. However it was important for even older systems that trusted DST Root CA X3 but were not aware about the ISRG Root X1 existence.

After September 30th 2021 those extremely old systems can not connect to servers using LE certificates and even including ISRG Root X1 in the server cert chain can not help them. Because DST Root CA X3 is dead for everyone.

However expiration of the DST Root CA X3 cert revealed another issue. Old software using old openssl versions refuses to connect to DA web server even when they do have DST Root CA X3 trusted as root cert! This is because they see a reference to no longer valid DST Root CA X3 certificate. This issue is not present in newer library version but still affects older (but not very old systems) where openssl is old but the system would otherwise trust ISRG Root X1.

For those cases we have released an update to the `letsencrypt` package. The key change is that we no longer include the ISRG Root X1 certificate in web server responses. The response from server with upgraded `letsencrypt` package would look like:

Code:
# openssl s_client -connect example.net:2222
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = example.net
verify return:1
---
Certificate chain
 0 s:/CN=example.net
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
...

We expect the system to have ISRG Root X1 cert present in the system and only provide server certificate and intermediate certificate R3. This allows older (but not extremely old) systems to connect to DA using LE certificates successfully which would otherwise fail if DST Root CA X3 were present in the response.

To download the update please follow these steps:

Code:
# /usr/local/directadmin/custombuild/build update
# /usr/local/directadmin/custombuild/build letsencrypt
# /usr/local/directadmin/scripts/letsencrypt.sh request your.server.name.example.net

Essentially these steps will:
* Update custombuild installation to know about new package versions
* Download latest letsencrypt package (version 2.0.23)
* Re-issue your main server certificate (and omit ISRG Root X1 in the chain)

TL;DR
Upgrading letsencrypt v2.0.23 package via the custombuild can allow some moderately old systems to connect to DA which currently fails to do so. New systems works fine with or without this upgrade. For extremely old systems there is nothing that can be done (except upgrading them).
Thanks so much.
I tried this on 2 servers but it didn't help :(
They are still facing same issue.
 

fln

Administrator
Staff member
Joined
Aug 30, 2021
Messages
96
Sorry to hear that it did not help :(. To be more precise I was focused on systems or applications that uses OpenSSL 1.0.x. My playground for reproducing this issue was Debian 8 system (which has ISRG root cert and old OpenSSL 1.0.1lt).

I would like to point out that following the instructions above renews only the certificate for the main server name. All the clients that has LE certificates issues for their domains will have longer cert chains until their certificates are renewed either manually or automatically.

@Richard G, thanks for the hints. After letsencrypt.sh script is updated to version 2.0.23 we just need to trigger certificate renewal, the actual command to do it does not matter that much. Triggering a certificate update from web UI would work as well.

If reissuing certificates with a shorter chain helps for some other cases, please let us know here. So we could better understand which OS / app combinations are actually affected by this.
 

system-admin

Verified User
Joined
Aug 1, 2019
Messages
59
Sorry to hear that it did not help :(. To be more precise I was focused on systems or applications that uses OpenSSL 1.0.x. My playground for reproducing this issue was Debian 8 system (which has ISRG root cert and old OpenSSL 1.0.1lt).

I would like to point out that following the instructions above renews only the certificate for the main server name. All the clients that has LE certificates issues for their domains will have longer cert chains until their certificates are renewed either manually or automatically.

@Richard G, thanks for the hints. After letsencrypt.sh script is updated to version 2.0.23 we just need to trigger certificate renewal, the actual command to do it does not matter that much. Triggering a certificate update from web UI would work as well.

If reissuing certificates with a shorter chain helps for some other cases, please let us know here. So we could better understand which OS / app combinations are actually affected by this.

Thanks again for your reply.

Our Server Environment:

CentOS 7.x / 64 Bit
DA Latest version

Client Side where issue is faced:

Windows 7 64 Bit
Chrome 94.x

Letsscript SSL works fine on Windows 7 with Firefox latest version but not with Chrome 94x
We have several clients using above environment.
 

sufiyanshaikh

Verified User
Joined
Aug 14, 2019
Messages
144
Hello,

My customers are getting "Warning" when they are trying to connect to their website. (almost 30% have complained)

Any possible solution to fix?
facing the same issue on same environment
Client Side where issue is faced:

Windows 7 64 Bit
Chrome 94.x

Letsscript SSL works fine on Windows 7 with Firefox latest version but not with Chrome 94x
We have several clients using above environment.
Same condition
 
Top