Apache client certificate validation, mutual TLS

[sec-is]

I am having a hard time getting this to work.
To set up a (sub) domain which will ask the end user for a certificate.
(I think) I know what is needed.
* I created a pem file containing the 4 certificates which I received. No extra spaces and such, and ordered from top to root (I know the order is not important, but still I did). I granted the file directadmin.access and debugging shows Apache reads the file correctly
* The O.S. is fully up to date, even rebooted to have everything in effect.
* The lower chain (Root and upwards) have even been added using update-ca-certificates (Debian) and the text showed "Updating certificates in /etc/ssl/certs..." and "3 added, 0 removed; done.".

I have a list of common ciphers, and tested from een external source these ciphers are in effect. They are okay.

In the virtualhost in DA I added:
SSLVerifyClient require
SSLVerifyDepth 4
SSLCACertificateFile /usr/local/directadmin/data/users/username/domains/mutual.testdomein.nl.pem

I did some debugging in Apache and here is a small part of it (mostly anonymized):
AH01914: Configuring server www.testdomein.nl:443 for SSL protocol
ssl_engine_init.c(638): Creating new SSL context (protocols: TLSv1.2, TLSv1.3)
ssl_engine_init.c(931): Configuring client authentication
ssl_engine_init.c(2212): AH02209: CA certificate: CN=001.hoofddomein.nl,serialNumber=00000007230463554300000,O=CENSOR,C=NL
ssl_engine_init.c(2212): AH02209: CA certificate: CN=QuoVadis PKIoverheid Private Services CA - G1,organization Identifier=NTRNL-112233,O=QuoVadis Trustlink B.V.,C=NL
ssl_engine_init.c(998): Configuring permitted SSL ciphers [ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305HE-RSA-AES256-GCM-SHA384HE-RSA-AES128-GCM-SHA256:!aNULL:!eNULL:!EXP]
ssl_engine_init.c(526): AH01893: Configuring TLS extension handling
ssl_util_ssl.c(436): [www.testdomein.nl:443] modssl_X509_match_name: expecting name 'www.testdomein.nl', matched by ID '*.testdomein.nl'
ssl_util_ssl.c(451): AH02412: [www.testdomein.nl:443] Cert matches for name 'www.testdomein.nl' [subject: CN=testdomein.nl / issuer: CN=R3,O=Let's Encrypt,C=US / serial: 035A3A88EE07FCBA9BA7E70E47DE22FEAE08 / notbefore: Nov 7 11:41:44 2023 GMT / notafter: Feb 5 11:41:43 2024 GMT]
AH02568: Certificate and private key www.testdomein.nl:443:0 configured from /etc/httpd/conf/domains/testdomein.nl.cert.combined and /etc/httpd/conf/domains/testdomein.nl.key
ssl_engine_init.c(1962): AH02556: "SSLOpenSSLConfCmd DHParameters /etc/httpd/conf/ssl.crt/dhparams.pem" applied to www.testdomein.nl:443

*NOTES: (1) you only see 2 out of the 4 certificates, Apache is possibly truncating the output. (2) the ciphers show some NULL values, I have no idea why but it would not be a problem since only the common ciphers are important.

I end up hearing back from the end user it did not work (they use a python based script and the error messages are not very useful). On my end all I (mostly) see is:
AH02261: Re-negotiation handshake failed
SSL Library Error: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed

I tried and looked at much more debug information, however all that does is show me the certificates being send and received, and all in hex notation, not very useful.

Something as simple as this, I do not understand why it is not working. I expect one setting somewhere in the chain of config files is doing this. I do not understand why re-negotiating is going on. Messages in the Apache log may differ a bit, since I tried different setups, those two are most often the ones visible.

In mean while I am requesting to get a test-certificate, not sure if I am going to get one. It wil make testing and debugging much faster. Till then, I can only do one or two tests a day.

 

[sec-is]

After several days of trying out many things I decided to try it on a different O.S. and it worked immediately.
So thank you Debian for not solving this problem. I found similar or exact the same problems in 2013 and 2017 and only on Debian systems (they seem related to http 2.0 for some reason).

I tested CentOs 7 and AlmaLinux and these work out of the box.

One more detail. I had placed the certificate in PEM format (the text format "BEGIN CERTIFICATE" thing) in
/usr/local/directadmin/data/users/username/domains/mutual.testdomein.nl.pem
and I referred to it
SSLCACertificateFile /usr/local/directadmin/data/users/username/domains/mutual.testdomein.nl.pem

The certificate holds 4 individual certificates, and in the debug info I only saw 2 being read by Apache. Very strange!
(the permissions were okay).
This is the line for debugging ssl: Loglevel ssl:debug
There are several more, dyor.

Then I made a copy of the quad-certificate to
/etc/httpd/conf/mutual.testdomein.nl.pem and referred to it.


This time I saw all 4 certificates in the debug output (Apache main site error log, /var/log/httpd/error_log ).

ALSO NOTE:
I had the quad-cert here:
/usr/local/directadmin/data/users/username/domains/mutual.testdomein.nl.pem

I did a backup transfer from DA to the new server.
I did not check, and had to find out that the file was not included in the backup. Or not restored, one of both.
So after the restore Apache would not start up, telling me that file was missing. I know it is not relevant (anymore) but still an important detail.

 

[sec-is]

One more detail. I had placed the certificate in PEM format (the text format "BEGIN CERTIFICATE" thing) in
/usr/local/directadmin/data/users/username/domains/mutual.testdomein.nl.pem
and I referred to it
SSLCACertificateFile /usr/local/directadmin/data/users/username/domains/mutual.testdomein.nl.pem

The certificate holds 4 individual certificates, and in the debug info I only saw 2 being read by Apache. Very strange!
(the permissions were okay).
This is the line for that:
Loglevel ssl:debug

Then I made a copy of the quad-certificate to
/etc/httpd/conf/mutual.testdomein.nl.pem and referred to it.


This time I saw all 4 certificates in the debug output.

ALSO NOTE:
I had the quad-cert here:
/usr/local/directadmin/data/users/username/domains/mutual.testdomein.nl.pem

I did a backup transfer from DA to the new server.
I did not check, and had to find out that the file was not included in the backup. Or not restored, one of both.
So after the restore Apache would not start up, telling me that file was missing. I know it is not relevant (anymore) but still an important detail.

 

[sec-is]

I leave my text above for others to find via a search engine.

Case closed.