Apache Down After Update to 2.4.59 - URGENT

Indeed adding more options, works really well in a lot of situations, but it can also make software almost dead when overused (or used when it should not have been used). It is not the complexity of adding such option that kills the software, but complexity of maintaining the option functionality when extending or refactoring software in the future.

Using feature flags is great when the side-effects could be localized. However we want to fundamentally change how CB works and maintenance for backwards compatibility in this particular case would make it extremely expensive. This change is the building block for a series of followup changes in how DA manages server certificates. Anyway this could easily lead to a very long discussion if we get into details.

Thanks for the feedback. Would be great to see if there are more scenarios (like one reported by @ccto) when certificate synchronization is not wanted. This could be a justification to introduce an opt-in CB flag to ignore TLS certificate synchronization when re-configuring services.
 
Quick update, certificate synchronization logic is updated to further reduce the possibility of accidental miss-configuration.

Combined certificate will be a concatenation of conf/cacert.pem and conf/carootcert.pem (at the time when certificates are synced) instead of direct copy of conf/cacert.pem.combined.

Updated logic looks like this

Code: 
       install_file "${key}"                                     /etc/httpd/conf/ssl.key/server.key          600 root:root || return 1
       install_file "${cert}"                                    /etc/httpd/conf/ssl.crt/server.crt          600 root:root || return 1 # Not really used in configs
       install_file "${ca_cert}"                                 /etc/httpd/conf/ssl.crt/server.ca           600 root:root || return 1 # Not really used in configs
       install_file <(cat "${cert}" <(printf "\n") "${ca_cert}") /etc/httpd/conf/ssl.crt/server.crt.combined 600 root:root || return 1

which is roughly equivalent to:

Code: 
          cp /usr/local/directadmin/conf/cakey.pem                                                /etc/httpd/conf/ssl.key/server.key
          cp /usr/local/directadmin/conf/cacert.pem                                               /etc/httpd/conf/ssl.crt/server.crt           # Not really used in configs
          cp /usr/local/directadmin/conf/carootcert.pem                                           /etc/httpd/conf/ssl.crt/server.ca            # Not really used in configs
          cat /usr/local/directadmin/conf/cacert.pem /usr/local/directadmin/conf/carootcert.pem > /etc/httpd/conf/ssl.crt/server.crt.combined

This makes sure it is not possible to have conflicting (different) certificates in conf/cacert.pem conf/carootcert.pem and conf/cacert.pem.combined files. The file conf/cacert.pem.combined is not actually used anymore.
 
fln said:
Quick update, certificate synchronization logic is updated to further reduce the possibility of accidental miss-configuration.

Combined certificate will be a concatenation of conf/cacert.pem and conf/carootcert.pem (at the time when certificates are synced) instead of direct copy of conf/cacert.pem.combined.

Updated logic looks like this

Code: 
       install_file "${key}"                                     /etc/httpd/conf/ssl.key/server.key          600 root:root || return 1
       install_file "${cert}"                                    /etc/httpd/conf/ssl.crt/server.crt          600 root:root || return 1 # Not really used in configs
       install_file "${ca_cert}"                                 /etc/httpd/conf/ssl.crt/server.ca           600 root:root || return 1 # Not really used in configs
       install_file <(cat "${cert}" <(printf "\n") "${ca_cert}") /etc/httpd/conf/ssl.crt/server.crt.combined 600 root:root || return 1

which is roughly equivalent to:

Code: 
          cp /usr/local/directadmin/conf/cakey.pem                                                /etc/httpd/conf/ssl.key/server.key
          cp /usr/local/directadmin/conf/cacert.pem                                               /etc/httpd/conf/ssl.crt/server.crt           # Not really used in configs
          cp /usr/local/directadmin/conf/carootcert.pem                                           /etc/httpd/conf/ssl.crt/server.ca            # Not really used in configs
          cat /usr/local/directadmin/conf/cacert.pem /usr/local/directadmin/conf/carootcert.pem > /etc/httpd/conf/ssl.crt/server.crt.combined

This makes sure it is not possible to have conflicting (different) certificates in conf/cacert.pem conf/carootcert.pem and conf/cacert.pem.combined files. The file conf/cacert.pem.combined is not actually used anymore.
Click to expand...
I'd still like to understand this, if I may, and this might help you avoid issues with other customers like me.

My initial problem was that Apache wouldn't start because server.ca cannot be empty.

You copied this file here:

/usr/local/directadmin/conf/carootcert.pem

But this file is empty. How / when does this file get updated by DA? Obviously, it should not be empty, and had it been complete I wouldn't have had this issue.

If the answer is that I have to enable Let's Encrypt, then I refer you to this page in your documentation:

Let's Encrypt For Domains | Directadmin Docs

DirectAdmin Knowledge Base
docs.directadmin.com

Let's Encrypt is an optional feature, setting letsencrypt=1 in directadmin.conf. I don't have this on because I don't want it on, I have Let's Encrypt working perfectly already.

But if Let's Encrypt is required to populate carootcert.pem, and you're copying that file (empty or not) all over the system, then this seems like a problem.

If my theory is completely incorrect, why would my carootcert.pem be empty? I haven't customised anything to make it this way.

Finally, I would like to make the point that Custombuild is exactly that - custom. The whole point of it is to give power to the server admin to customise their server.

The command is "build apache". Nowhere in that command does it suggest that SSL files will be overwritten. This is a significant change in behaviour, and the behaviour is not implied by "build apache". Otherwise it should be "build apache and rewrite my SSL certificates".

This is solved by putting a variable in options.conf.
 
Richpark said:
/usr/local/directadmin/conf/carootcert.pem

But this file is empty. How / when does this file get updated by DA? Obviously, it should not be empty, and had it been complete I wouldn't have had this issue.
Click to expand...

This is created in one of the following ways:
  • If DA runs in plain-text mode it will be an empty file
  • If DA runs SSL mode
    • when letsencrypt integration is used it will be created by letsencrypt.sh
    • when cert is installed manually it should be created by server admin
This file is supposed to contain chain (intermediate) certificates. Not the server cert (which should be in cacert.pem) and not root CA cert (which gets bundled with browsers or OS) but all the intermediate CA certificates that ensures that there is a signature chain from root CA cert which browser trusts up to server cert.

It is OK for chain certificate to be empty if server cert is issued directly from root CA or if server cert file contains chain certificates. For example if self signed certificates are used it is perfectly fine to have empty chain certificate file.

Richpark said:
If my theory is completely incorrect, why would my carootcert.pem be empty? I haven't customised anything to make it this way.
Click to expand...
If you are running DirectAdmin in TLS mode (and it works fine), then I think you have placed your chain certificates inside the cacert.pem file. It is not correct thing to do in theory but in practice it does not matter at all since all services supports loading both main and chain cert from single file (it is mostly a side effect of services using openssl for TLS).

The root of the problem is your custom apache configuration. Long time ago it apache used to load main cert and chain certificates using different directives - SSLCertificateFile should point to server cert and SSLCertificateChainFile would point to chain certs. Starting openssl 1.0.2 and apache 2.4.8 chain file is obsolete since apache can load both server cert and chain certs from a single file. This is the reason why CustomBuild does not use the SSLCertificateChainFile directive anymore and uses concatenated version. The files are copied for backwards compatibility with old CB/apache if multiple cert files were used.

It seems in your customized apache configuration you have used SSLCACertificateFile "/etc/httpd/conf/ssl.crt/server.ca". The SSLCACertificateFile directive is supposed to tell apache what root CA certificates to use when validating client certs. Normally web browsers (clients of webserver) does not authenticate themselves using TLS. They do check for webserver to have a valid certificate, but server does not ask the client to provide a certificate. TLS can be used for mutual authentication where client could install a certficate to the browser and use this certificate to identify itself when connecting to the servers. This is rarely used configuration, mostly not by browsers but for HTTP based server to server communications when both parties wants to authenticate one another.

So unless you want to authenticate your webserver clients you should not even have the SSLCACertificateFile directive in the config.

My guess would be that you have mixed up SSLCACertificateFile with SSLCertificateChainFile because previously CustomBuild used to install chain certificate into the /etc/httpd/conf/ssl.crt/server.ca file.

Richpark said:
The command is "build apache". Nowhere in that command does it suggest that SSL files will be overwritten. This is a significant change in behaviour, and the behaviour is not implied by "build apache". Otherwise it should be "build apache and rewrite my SSL certificates".
Click to expand...
Command build apache used to mean - reinstall apache by recompiling it, rewrite all configuration files to be up to date (fix them if they were messed up), ensure apache - DA integration is working (adding a domain via DA affects apache configuration).

It used to do many things, it would even create you a self-signed certificate if you have no certificates at all (running DA in plain-text mode), it if you run DA in TLS mode it would copy the certificates when you are building apache for the first time but from that point forward it would not update certs if they are invalid.

Since DA 1.662 in addition to all the stuff CB does it also copies the certificates to make sure DA and apache uses same host certificate. Keep in mind this whole post is about server hostname certificates (used when accessing the server by server hostname or IP), not certificates for user domains.
I have tried to be concise without going into too much details, hope this helps. Let me know if you need clarification on some points.
 
fln said:
The root of the problem is your custom apache configuration. Long time ago it apache used to load main cert and chain certificates using different directives - SSLCertificateFile should point to server cert and SSLCertificateChainFile would point to chain certs. Starting openssl 1.0.2 and apache 2.4.8 chain file is obsolete since apache can load both server cert and chain certs from a single file. This is the reason why CustomBuild does not use the SSLCertificateChainFile directive anymore and uses concatenated version. The files are copied for backwards compatibility with old CB/apache if multiple cert files were used.

It seems in your customized apache configuration you have used SSLCACertificateFile "/etc/httpd/conf/ssl.crt/server.ca". The SSLCACertificateFile directive is supposed to tell apache what root CA certificates to use when validating client certs. Normally web browsers (clients of webserver) does not authenticate themselves using TLS. They do check for webserver to have a valid certificate, but server does not ask the client to prevent a certificate. TLS can be used for mutual authentication where client could install a certficate to the browser and use this certificate to identify itself when connecting to the servers. This is rarely used configuration, mostly not by browsers but for HTTP based server to server communications when both parties wants to authenticate one another.

So unless you want to authenticate your webserver clients you should not even have the SSLCACertificateFile directive in the config.
Click to expand...
Thank you for taking the time to reply in this detail. The information is very helpful, and I do understand everything you have said in this post.

I want to correct one thing, however - I haven't modified the default "httpd-ssl.conf" file that comes with CustomBuild, this one:

/usr/local/directadmin/custombuild/configure/ap2/conf/extra/httpd-ssl.conf

I'm looking at this file now, it's completely untouched by me, provided by CustomBuild, latest version. It contains the following uncommented lines:

Code: 
SSLCertificateFile /etc/httpd/conf/ssl.crt/server.crt
SSLCertificateKeyFile /etc/httpd/conf/ssl.key/server.key
SSLCACertificateFile /etc/httpd/conf/ssl.crt/server.ca

The third line is the one you're telling me should not be used. There seems to be an acceptance that I have heavily customised my Apache config, but I haven't. This is the original file, here's the full block relating to that line:

Code: 
#   Certificate Authority (CA):
#   Set the CA certificate verification path where to find CA
#   certificates for client authentication or alternatively one
#   huge file containing all of them (file must be PEM encoded)
#   Note: Inside SSLCACertificatePath you need hash symlinks
#         to point to the certificate files. Use the provided
#         Makefile to update the hash symlinks after changes.
SSLCACertificateFile /etc/httpd/conf/ssl.crt/server.ca
#SSLCACertificatePath /etc/httpd/conf/ssl.crt
#SSLCACertificateFile /etc/httpd/conf/ssl.crt/ca-bundle.crt

If I've misunderstood, my apologies. Maybe a setting somewhere in DA produces an alternate version of this file? But it doesn't seem to correlate with what you've said above.
 
Ohh I see, sorry I just assumed this is all due to customizations, 9 out of 10 reported problems are caused by customizations. If you have no apache customizations then this changes things completely.

I got back to check what gets bundled with CustomBuild and indeed extra/httpd-ssl.conf template file does contain NOT commented-out SSLCACertificateFile directive.

It seems the incorrect interpretation on what SSLCACertificateFile does was made not by you but by DA/CustomBuild long time ago. Checking the CB code I found CustomBuild auto-corrects this issue by commenting out this field during installation. This automatic fixing code is conditional and designed to only affect new DA installations that started using combined certificates. Essentially not touching old systems and preserving them with old apache config.

Thanks a lot for bringing this to our attention ?‍♂️, really glad we got to the bottom if it! I was not aware of this break but then fix CustomBuild behaviour introduced long time ago ?. Since CB is now in control of both config file and cert file we will be able to clean things up.
 
I should start out by saying that I'm not using DirectAdmin v1.662 and Apache v2.4.59 yet. It's specifically for the reasons that are in this thread that I always hold off on DirectAdmin updates until I can glean where potential issues are with updates. Having said all of that, I'm still having trouble wrapping my head around what specifically is going on with this thread and probably will until I peform an upgrade and see this for myself. But at least I know that this can be a potential issue.

Richpark said:
The default DA file here:

/usr/local/directadmin/custombuild/configure/ap2/conf/extra/httpd-ssl.conf

Doesn't have that line commented out, so I don't know why yours is.
Click to expand...

To me this is the crux of the issue (or "a" issue). Why are /usr/local/directadmin/custombuild/configure/ap2/conf/extra/httpd-ssl.conf and /etc/httpd/conf/extra/httpd-ssl.conf different? Where is /etc/httpd/conf/extra/httpd-ssl.conf being built from? As far as I know, I don't have anything customized to answer this. And if that were the case, then others in this thread would seem to be using the same customization. /etc/httpd/conf/extra/httpd-ssl.conf has SSLCACertificateFile /etc/httpd/conf/ssl.crt/server.ca comment out, /usr/local/directadmin/custombuild/configure/ap2/conf/extra/httpd-ssl.conf has SSLCACertificateFile /etc/httpd/conf/ssl.crt/server.ca active.



I assume all of this stems from DirectAdmin v1.662's update:

Version 1.662 | Directadmin Docs

DirectAdmin Knowledge Base
docs.directadmin.com

And the fact that this is not a togglable option is disappointing.

My intent isn't to be overly critical here, but constructive criticism is valuable.

Much like @Richpark - I too run my own Let's Encrypt certificate issuance system. My story goes that I created this system for another control panel before they released their own system and then when I started with DirectAdmin I just integrated that system into DirectAdmin. Anybody that programs - at least at this scale - will probably tell you, they prefer their own system rather than someone else's. If there's an issue with my system... I get to fix it, which means I can fix it on my timetable. When you're constantly depending on someone else's system, then you have to depend on their timetable for fixing issues. That's not a knock against people who depend on other's systems, it's just meant to reveal why people who design their own systems often prefer their own systems.

For me this always comes down to the question: Is a web hosting control panel meant to be a tool for server administrators or is it meant to be an all-encompassing self-contained web hosting management system?

How you answer that question will likely decide your stance on this change in DirectAdmin v1.662.

I fear that this change is a step towards DirectAdmin wanting to become an all-encompassing self-contained web hosting management system. And this may be, or probably is, what the mass of DirectAdmin license holders want: Install DirectAdmin and sit back and watch it work. I've already voiced an opinion against tying stack application updates to DirectAdmin updates and this change seems to go hand in hand with this move.

Those of us who see a control panel as a tool for server administration probably prefer a bit more compartmentalization: Web server, MTA, MDA, Database server, PHP, certificate issuance, and control panel - all work largely independently of each other. There does have to be some interoperability between the elements - i.e. PHP has to be compiled with support for the database server in use on the server (although, technically database isn't a requirement for PHP) - but largely one element doesn't depend on specifics of the other elements.

In this way, the web server being upgraded doesn't depend on any other elements being upgraded.

A quick aside, may be off-topic... but may also help to round this into form. Let's Encrypt is not a web server (Apache), MTA (Exim), or MDA (Dovecot) thing. Let's Encrypt is a certificate authority. You get certificates from a certificate authority, what you do with those certificates is another process. The ACME protocol is a system from which you can automatically request a certificate from a certificate authority (i.e. Let's Encrypt). The ACME protocol doesn't have anything to do with installing requested certificates into your front-end systems. Control panels seem to like to automate this into an all-in-one - and largely that's what the web hosting industry wants to do. But if you break this up into its individual parts then you gain more control of how the system reacts. This is why I differentiate between certificate installation and certificate issuance. You have to request->get issued->retrieve a certificate from a certificate authority (or a self-signed certificate) before you can install the certificate anywhere.

This is how I wrote my system. It requests and gets issued a certificate for a given domain name (plus SANs) and when retrieved - I then have a certificate. From there I can use the API system (Thanks DirectAdmin!) of the control panel to install the certificate for the domain name for the given services. The certificate issuance system is monolithic. I can use that system on any control panel. It's installing the certificate that depends on the individual web server, MTA, MDA - or if there is a control panel API for installing these.

Again, I don't use DirectAdmin's built-in certificate issuance and installation system so I don't really know how it works, but I suspect that it's doing a similar thing all at once. But you sacrifice control with this system because you are walled into operating exactly how DirectAdmin does this. And that's fine, I'm not meaning to be critical of that. However I am hoping to illustrate the difference between using DirectAdmin (or any control panel) as a tool versus using the control panel as a full self-administration system.

The fact that DirectAdmin is wanting to sync certificates after every web server rebuild breaks that compartmentalization. Installing or upgrading a web server doesn't have anything to do with the certificates in use. If the argument is that service certificates get mangled up to where they are not always in sync... then that's an issue with the server administrator. The system they are using to issue and install certificates isn't doing proper clean-up or maybe they specifically want it that way. It is my belief that it's not DirectAdmin's responsibility to decide this fate. Now if DirectAdmin wants to add a togglable post-web server install/upgrade hook that performs this synchronization, I'd be OK with that - and maybe that helps a lot of users. But that doesn't appear to be what is happening. Instead of seeing each service as its own independent system, it's saying that it owns the whole ecosystem and can do with it however it pleases. I think that's a dangerous road to head down.
 
fln said:
template file does contain NOT commented-out SSLCACertificateFile directive.
Click to expand...
Correct, but for some reason the /etc/httpd/conf/extra/httpd-ssl.conf file does have it commented out. I always thought that, without customisation, the template file was copied 1:1 to the destination. Seems not the case with this one. Or am I missing something?

Edit: I was writing at the same time as @sparek, so didn't see he already mentioned this too.
 
An update is released for CustomBuild apache configuration files to match more closely what is actually used. Previous versions would have config files with SSLCACertificateFile directive uncommented, but then make sure it is commented-out after file is copied to /etc/httpd. With this change files in ./custombuild/configure/ap2 will match more closely with /etc/httpd. It still does not match 100% because CustomBuild needs to update IP and PORT fields, but it will not longer mangle TLS configuration.
 
fln said:
An update is released for CustomBuild apache configuration files to match more closely what is actually used.
Click to expand...
I hope this "update" wont harm or break other users who simply using DA just "out of the box" without much customation
 
Last edited:
sparek said:
It is my belief that it's not DirectAdmin's responsibility to decide this fate. Now if DirectAdmin wants to add a togglable post-web server install/upgrade hook that performs this synchronization, I'd be OK with that - and maybe that helps a lot of users. But that doesn't appear to be what is happening. Instead of seeing each service as its own independent system, it's saying that it owns the whole ecosystem and can do with it however it pleases. I think that's a dangerous road to head down.
Click to expand...
I couldn't agree more. It looks like there will need to be more voices than just yours and mine asking for this, however. People need to speak up if they want to maintain the granular control they've always had in DA.

For now, I always approach updates in the same way - I have an identical test VM that I use first, and if there are problems I resolve them on there before I roll out to my production VMs.

This issue took two days out of my week, however, with all of the investiation, solution implementation, and back/forth discussion on here. I hope something like this doesn't happen again for a long time, it's just so disruptive.
 
You must log in or register to reply here.
Back
Top