Why is /etc/httpd/conf/ssl.crt/server.ca empty? (httpd fail to start)

[LowRadio]

Woke up to failed httpd email notifications
check whats going on
journalctl -xe

server.org systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
server.org kill[5219]: kill: cannot find process ""
server.org systemd[1]: httpd.service: control process exited, code=exited status=1
server.org systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has failed.

Doesn't help much, next try:
/var/log/httpd/error_log

AH00016: Configuration Failed
[Wed Jun 07 10:23:08.001930 2017] [ssl:emerg] [pid 5425:tid 139933368420416] AH02311: Fatal error initialising mod_ssl, exiting. See /var/log/httpd/domains/somedomain.com.error.log for more information

ok still need more info:
/var/log/httpd/domains/somedomain.com.error.log

AH01895: Unable to configure verify locations for client authentication
SSL Library Error: error:0B084009:x509 certificate routines:X509_load_cert_crl_file:PEM lib

Well it appears to be an SSL issue:
/usr/local/directadmin/data/users/someuser/httpd.conf

<VirtualHost 1.22.333.4:443>
        SSLEngine on
        SSLCertificateFile /usr/local/directadmin/data/users/someuser/domains/somedomain.com.cert
        SSLCertificateKeyFile /usr/local/directadmin/data/users/someuser/domains/somedomain.com.key
        SSLCACertificateFile /etc/httpd/conf/ssl.crt/server.ca   [COLOR="#FF0000"]<---(hmmm thats odd)[/COLOR]

cat /etc/httpd/conf/ssl.crt/server.ca

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

where did it go?
Find backup, restored the /etc/httpd/conf/ssl.crt/server.ca file.
systemctl start httpd.service
Success!


Check logs to see what happened shortly before httpd start fail:
/var/log/directadmin/system.log

2017:06:07-00:10:29: LetsEncrypt(8797): /usr/local/directadmin/scripts/letsencrypt.sh renew 'server.org' 4096 /usr/local/directadmin/conf/ca.san_config /var/www/html
2017:06:07-00:12:44: LetsEncrypt(8797): exit code: 0

Not sure if that was the caused /etc/httpd/conf/ssl.crt/server.ca to become blank.
Any ideas what it could be?
Im running on Centos7 64
Apache 2.4.23 with SNI and HTTP2 enabled
everything else is up to date.

 

[zEitEr]

Hello,

Is the script letsencrypt.sh updated? Did you try to run manually the letsencrypt.sh renew command from the logs? What results did you have?

 

[LowRadio]

Hello,

Is the script letsencrypt.sh updated?

letsencrypt.sh #VERSION=1.0.10

Did you try to run manually the letsencrypt.sh renew command from the logs? What results did you have?

I think I did try to run it before restoring /etc/httpd/conf/ssl.crt/server.ca and it failed because apache would not start.
However, after I restored it, I did run letsencrypt.sh manually to update the hostserver cert (it was due for an update anyway) and it worked as it should.

I don't have a /var/log/letsencrypt.log any idea where I could find more detailed logs?

 

[zEitEr]

The lines from /var/log/directadmin/system.log included a command

/usr/local/directadmin/scripts/letsencrypt.sh renew 'server.org' 4096 /usr/local/directadmin/conf/ca.san_config /var/www/html

actually I was refering to it, and you already answered.

Do you have custom templates for virtual hosts of Apache? The line SSLCACertificateFile /etc/httpd/conf/ssl.crt/server.ca should not specify server.ca there.

Probably it is too late, but I would check last modification time of the file /etc/httpd/conf/ssl.crt/server.ca and then would check cron logs in order to find what was running at that time.

 

[LowRadio]

Do you have custom templates for virtual hosts of Apache? The line SSLCACertificateFile /etc/httpd/conf/ssl.crt/server.ca should not specify server.ca there.

No custom template.
The line in /usr/local/directadmin/data/users/someuser/httpd.conf that pointed to SSLCACertificateFile /etc/httpd/conf/ssl.crt/server.ca was changed because in DA, USER section I switched from a letsencrypt cert to a localcert hoping the would fix the apache startup failure.
I added that to my post just to show how I found the corrupt ssl.crt/server.ca file.
Once apache was running I switched back to the letsencrypt cert.

I also tried to disable ssl for that user's domain but apache would fail anyway and give me a similar error but for the next domain on the list and so on.

Probably it is too late, but I would check last modification time of the file /etc/httpd/conf/ssl.crt/server.ca and then would check cron logs in order to find what was running at that time.

yeah it is too late but server.crt.backup had a time stamp

-rw-------  1 root root 2151 Jun  7 00:12 server.crt.backup

I checked the cron logs and /var/log/messages. Can't see anything of importants.

Jun  7 00:06:18 gce run-parts(/etc/cron.hourly)[8720]: finished 0yum-hourly.cron
Jun  7 00:07:01 gce CROND[8737]: (root) CMD (/usr/local/directadmin/dataskq)
Jun  7 00:08:01 gce CROND[8768]: (root) CMD (/usr/local/directadmin/dataskq)
Jun  7 00:09:01 gce CROND[8782]: (root) CMD (/usr/local/directadmin/dataskq)
Jun  7 00:10:01 gce CROND[8796]: (root) CMD (echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue)
Jun  7 00:10:01 gce CROND[8797]: (root) CMD (/usr/local/directadmin/dataskq)
Jun  7 00:11:01 gce CROND[10848]: (root) CMD (/usr/local/directadmin/dataskq)
Jun  7 00:12:01 gce CROND[10867]: (root) CMD (/usr/local/directadmin/dataskq)
Jun  7 00:13:01 gce CROND[11267]: (root) CMD (/usr/local/directadmin/dataskq)

I'm thinking its an issue with letsencrypt

https://letsencrypt.status.io/pages/history/55957a99e800baa4470002da

June 8, 2017 6:10PM MDT
June 9, 2017 12:10AM UTC
[Investigating] We are investigating a disruption to certificate issuance.