Apache Down After Update to 2.4.59 - URGENT

KevinFlex said:
We have the same issue on several servers so far, I would like to know the root cause aswell in order to fix it.
Click to expand...
It’s a relief to hear that we’re not alone, but I’m sorry to hear you have this problem as well. The server can be restarted by manually replacing the server.ca and combined files as described above, but every time Apache is built they are zeroed again.

DA needs to fix this, rebuilding Apache shouldn’t overwrite the existing SSL files AT ALL, for any reason. I don’t believe it used to do this.

Does anyone have a support contract who can raise this issue?
 
Richard G said:
You still have an issue, because it shouldn't be in there, I will write to you via pm to not disturb the topic further.
Click to expand...
KevinFlex said:
We have the same issue on several servers so far, I would like to know the root cause aswell in order to fix it.
Click to expand...
Well, sometimes i need sleep and not to be on the pc... hahaha

/usr/local/directadmin/custombuild/configure/ap2/conf/extra/httpd-ssl.conf
Click to expand...
has the line
SSLCACertificateFile /etc/httpd/conf/ssl.crt/server.ca --> with no #

/etc/httpd/conf/extra/httpd-ssl.conf
Click to expand...
Has a # in front of it at the same line.

/etc/httpd/conf/ssl.crt/server.ca
Click to expand...

Is a correct file with unreadable text. (certificate)
 
Hi Guys,

we've had one server falling out this morning with this issue - and solved via this thread. Many thanks.

My question is; would disabling auto updates by DA prevent this? (https://docs.directadmin.com/change...guration-option-autopatch-in-directadmin-conf) We have a manual update calender which touches every server once a month outside officehours. So disabling auto updates (keeping hotfixes still on auto) would give us peace of mind and control over eventual downtime.

Can someone confirm this?
Thanks!
 
The change in behaviour is cause by CustomBuild installing default server certificate to the services.

For default server TLS certificate (not user owned domain certificates) CB copies them from:
  • /usr/local/directadmin/conf/cakey.pem - key file
  • /usr/local/directadmin/conf/cacert.pem - certificate file
  • /usr/local/directadmin/conf/carootcert.pem - certificate chain file
  • /usr/local/directadmin/conf/cacert.pem.combined - combined main certificate and chain certs
If you are manually managing the certificates make sure they are installed in those locations (unless default locations are changed in directadmin.conf) and CB will install those certificates to the services it builds.

Previously CB used to completely ignore the certificate existence, with one exception - generating self signed certs if they are missing.

The intention of this change is to have a single place as source of truth for server certificate. It will be used by DA and by all services that CB manages (apache, nginx, litespeed, openlitespeed, exim, dovecot, proftpd, pureftpd). So using custom certificate flow is:
  1. Install certificate for DA
  2. Run da build sync_server_cert to distribute it to other services.
 
What CB does with sync_server_cert command is:

Code: 
       install_file "${key}"                                     /etc/httpd/conf/ssl.key/server.key          600 root:root || return 1
       install_file "${cert}"                                    /etc/httpd/conf/ssl.crt/server.crt          600 root:root || return 1 # Not really used in configs
       install_file "${ca_cert}"                                 /etc/httpd/conf/ssl.crt/server.ca           600 root:root || return 1 # Not really used in configs
       install_file <(cat "${cert}" <(printf "\n") "${ca_cert}") /etc/httpd/conf/ssl.crt/server.crt.combined 600 root:root || return 1

which is roughly equivalent to:

Code: 
          cp /usr/local/directadmin/conf/cakey.pem                                                /etc/httpd/conf/ssl.key/server.key
          cp /usr/local/directadmin/conf/cacert.pem                                               /etc/httpd/conf/ssl.crt/server.crt           # Not really used in configs
          cp /usr/local/directadmin/conf/carootcert.pem                                           /etc/httpd/conf/ssl.crt/server.ca            # Not really used in configs
          cat /usr/local/directadmin/conf/cacert.pem /usr/local/directadmin/conf/carootcert.pem > /etc/httpd/conf/ssl.crt/server.crt.combined

Same actions are performed when reinstalling apache.

Note: This post is edited with updated combined certificate copying logic.
 
Last edited:
fln said:
The change in behaviour is cause by CustomBuild installing default server certificate to the services.

For default server TLS certificate (not user owned domain certificates) CB copies them from:
  • /usr/local/directadmin/conf/cakey.pem - key file
  • /usr/local/directadmin/conf/cacert.pem - certificate file
  • /usr/local/directadmin/conf/carootcert.pem - certificate chain file
  • /usr/local/directadmin/conf/cacert.pem.combined - combined main certificate and chain certs
If you are manually managing the certificates make sure they are installed in those locations (unless default locations are changed in directadmin.conf) and CB will install those certificates to the services it builds.

Previously CB used to completely ignore the certificate existence, with one exception - generating self signed certs if they are missing.

The intention of this change is to have a single place as source of truth for server certificate. It will be used by DA and by all services that CB manages (apache, nginx, litespeed, openlitespeed, exim, dovecot, proftpd, pureftpd). So using custom certificate flow is:
  1. Install certificate for DA
  2. Run da build sync_server_cert to distribute it to other services.
Click to expand...
My mind is blown that you can introduce a BREAKING change like this and not even make it a togglable option. You just do this and "assume" that everyone will be okay, or everyone uses the software the way you hope they do.

I'll take this opportunity to say that DA has gone massively downhill since John took a step back, or sold the business, I don't know what he did - but quality control has gone down the toilet during the last couple of years. I've started to dread every new DA update, it only breaks things.

How do you turn off this unwanted behaviour so I can continue to manage my own certificates?

It's not just what you've described above, it's ProFTP as well:

Code: 
Apr 16 12:22:39 *.co.uk systemd[1]: Starting ProFTPD FTP Server...
Apr 16 12:22:39 *.co.uk proftpd[28449]: Checking syntax of configuration file
Apr 16 12:22:39 *.co.uk proftpd[28449]: 2024-04-16 12:22:39,079 *.co.uk proftpd[28449]: fatal: TLSRSACertificateFile: '/etc/exim.cert' does not exist or does not contain a certificate on line 22 of '/etc/proftpd.conf'
Apr 16 12:22:39 *.co.uk systemd[1]: proftpd.service: control process exited, code=exited status=1
Apr 16 12:22:39 *.co.uk systemd[1]: Failed to start ProFTPD FTP Server.
Apr 16 12:22:39 *.co.uk systemd[1]: Unit proftpd.service entered failed state.
Apr 16 12:22:39 *.co.uk systemd[1]: proftpd.service failed.

So exim.cert has been zeroed as well, and that won't start either. I'm now afraid to update anything on any of our servers.

Admit you've made a hell of a mess and fix this please.
 
Yes. Certificates are installed the same way for other services as well. It was mentioned in the previous post.
fln said:
It will be used by DA and by all services that CB manages (apache, nginx, litespeed, openlitespeed, exim, dovecot, proftpd, pureftpd).
Click to expand...

If you are using CB to manage services configuration you can continue using your own certificates and your own way of getting certificates as long as you install them in DA config directory. CB will take care to propagate them further to other services.

This was mentioned in the change-log:
Host certificates from DirectAdmin config are always copied to the services, not the other way around. If some new certificate is manually installed for a particular service (for example Apache), but old certificate is left for DirectAdmin service, then calling da build sync_server_cert or da build rewrite_confs will copy the old cert from DirectAdmin to Apache.
Click to expand...

This is a breaking change for you, because you like for CB to manage configs but not manage certificates. But this is an improvement for the rest of DA users. This is because if you have messed up your apache configuration (or certificates) rebuilding apache will fix it. Previously it would not touch certificates and leave apache in a broken state.

Without this change manually managing certificates was quite error prone, because you could install new certificate for some services but not all of them. Since now CB takes care of managing service certs (treats certificates as part of service configuration) it is NOT now possible to have malformed configs (when some services use new certificate while other uses old certificate).

If you really do not want CB to get in your way you could just use different paths for you custom certificates. CB takes care to sync certificates to the fixed locations (where it expects them to be). Or you could take even one step further and not use CB to manage service configuration if you need to configure them drastically differently.

We will not be adding a configuration option for this.
 
fln said:
Yes. Certificates are installed the same way for other services as well. It was mentioned in the previous post.


If you are using CB to manage services configuration you can continue using your own certificates and your own way of getting certificates as long as you install them in DA config directory. CB will take care to propagate them further to other services.

This was mentioned in the change-log:


This is a breaking change for you, because you like for CB to manage configs but not manage certificates. But this is an improvement for the rest of DA users. This is because if you have messed up your apache configuration (or certificates) rebuilding apache will fix it. Previously it would not touch certificates and leave apache in a broken state.

Without this change manually managing certificates was quite error prone, because you could install new certificate for some services but not all of them. Since now CB takes care of managing service certs (treats certificates as part of service configuration) it is NOT now possible to have malformed configs (when some services use new certificate while other uses old certificate).

If you really do not want CB to get in your way you could just use different paths for you custom certificates. CB takes care to sync certificates to the fixed locations (where it expects them to be). Or you could take even one step further and not use CB to manage service configuration if you need to configure them drastically differently.

We will not be adding a configuration option for this.
Click to expand...
So servers such as mine and your customers above in this thread are all acceptable "collateral damage" for you? Do you understand that we have live servers with customers of our own who stop paying us or leave when our services go down?

You have built a system that copies "/usr/local/directadmin/conf/carootcert.pem" over the top of working files EVEN IF IT'S BLANK. I don't even know why it's blank, it's not something I've ever had to worry about. Does Let's Encrypt have to be activate via DA now for the control panel to function as expected?

You have customers who have been with you for a decade or more, you don't seem to care. Your post sounds extremely arrogant.

Wait for more of your customers' servers to go down, this is just the first batch of complaints.
 
Well, DA makes managing servers "more easy" thats why we use the product.
Some of us could setup the servers, maintain them fully without a GUI, some of us are lazy, some of us dont understand .... about how and what, and DA makes it possible for us to maintain our servers.

And DA will always implement features / improvements based on how they setup the services, they cant take note of every change people might make because then they are never able to update stuff....

You decided one day to change something outside of the scope of DA, than you have to note that down, read the change logs and then see if you would need to update, need to update and then change something etc.

Richpark said:
collateral damage
Click to expand...

He never spoke about it like that, he just said;

fln said:
If you are using CB to manage services configuration you can continue using your own certificates and your own way of getting certificates as long as you install them in DA config directory. CB will take care to propagate them further to other services.

This was mentioned in the change-log:
Click to expand...


So you are free to do whatever, but if you want to use CB, than sometimes something that you changed manually might break, and you will have to fix it.
 
Sorry if I sound arrogant but I really wanted to stress out this was done deliberately rather than by an accident.

Over the years DirectAdmin accumulated over 500 configuration options, and CustomBuild being heavily customizable created a lot of opportunities to achieve the same result in multiple ways. This makes it extremely hard to change anything without breaking some custom workflow. Our main goal over the last year was actually working towards simplifying and unifying DirectAdmin and CustomBuild configuration. And this means breaking stuff for users who used DA in a non standard way.

Our goal is to have single source of truth of what server certificate is. We know for sure this is a real problem for some users (having different certs for different services without an easy way to fix). We picked DA configuration to be the central location because all other services are optional. Most of the users either use LetsEncrypt or manage certs manually but they still want to install the certs for DA as well. So this approach should be ok for most of the users.

Now lets consider what would happen if we added a migration layer to try to guess where the correct certificate is (instead of blindly using certs from DA config):
  • You would not have noticed it and would continue installing certs for the service, so on the first run we would pick cert from apache config and pull it into DA config, but after you update your cert, the normal logic would kick in and would replace your new cert with whatever is in DA config where we pulled the old cert when we performed the migration.
  • Suppose someone has correct cert un Exim config, but different (bad but not empty) cert for Apache? Which one should we pick.
  • Suppose you have server cert, chain cert and combined cert each having different content, which one is the real one?
  • How can we know you are even using the files which are commented out in the default apache configuration? Other people might have junk or old certificates in place for those files not used by config. Now imagine the harm we could do if we treated them as source of truth and pulled is as DA cert, or even worse propagate them to other services.
Without breaking the configs we have no other way of saying "you are doing it wrong, please stop doing it this way and to it that way". Any automatic migration would make things even more hard to understand and would most likely break things at some later point in time.

After this change it is simple - whatever you have configured in DA will be used by all CB managed services. If something is wrong you will see this immediately and will need to adjust.
 
Last edited:
I do agree with @ericosman and can only reiterate - whenever you customize something you need to keep track the change-log and updates very carefully, as things might change. We can not say compatible with the stuff we do no know even exists :).

Richpark said:
And this sentence shows you don't know what you're talking about, so please stay out of it.
Click to expand...
Please, lets stay polite and respectful.
 
fln said:
Sorry if I sound arrogant but I really wanted to stress out this was done deliberately rather than by an accident.

Over the years DirectAdmin accumulated over 500 configuration options, and CustomBuild being heavily customizable created a lot of opportunities to achieve the same result in multiple ways. This makes it extremely hard to change anything without breaking some custom workflow. Our main goal over the last year was actually working towards simplifying and unifying DirectAdmin and CustomBuild configuration. And this means breaking stuff for users who used DA in a non standard way.

Our goal is to have single source of truth of what server certificate is. We know for sure this is a real problem for some users (having different certs for different services without an easy way to fix). We picked DA configuration to be the central location because all other services are optional. Most of the users either use LetsEncrypt or manage certs manually but they still want to install the certs for DA as well. So this approach should be ok for most of the users.

Now lets consider what would happen if we added a migration layer to try to guess where the correct certificate is (instead of blindly using certs from DA config):
  • You would not have noticed it and would continue installing certs for the service, so on the first run we would pick cert from apache config and pull it into DA config, but after you update your cert, the normal logic would kick in and would replace your new cert with whatever is in DA config where we pulled the old cert when we performed the migration.
  • Suppose someone has correct cert un Exim config, but different (bad but not empty) cert for Apache? Which one should we pick.
  • Suppose you have server cert, chain cert and combined cert each having different content, which one is the real one?
  • How can we know you are even using the files which are commented out in the default apache configuration? Other people might have junk or old certificates in place for those files not used by config. Now imagine the harm we could do if we treated them as source of truth and pulled is as DA cert, or even worse propagate them to other services.
Without breaking the configs we have no other way of saying "you are doing it wrong, please stop doing it this way and to it that way". Any automatic migration would make things even more hard to understand and would most likely break things at some later point in time.

After this change it is simple - whatever you have configured in DA will be used by all CB managed services. If something is wrong you will see this immediately and will need to adjust.
Click to expand...
As one programmer to another, don't you think you should have a line of code which says, if CA file is blank and copying it will prevent Apache from starting, don't copy the files?

I reiterate, these are live production servers with online businesses hosted on them. And if I'm angry, and it's showing in my posts, then I cannot apologise, I have every right to be angry. This change of yours has wasted 2 days of my working week. If you can't empathise with my situation, then I think it's sad.
 
IMHO -

No matter it is a valid certificate or self-signed certificate , changing Apache SSL certificate, FTP SSL certificate are of little impact.

---

However, we are very cautious when changing Exim (and Dovecot) default SSL (i.e. /etc/exim.key and /etc/exim.cert)

From our experience, for domain without SSL, those users will use /etc/exim.key and /etc/exim.cert automatically.

For those iOS devices, when successfully configured POP/IMAP/SMTP with self-signed certificate in the past, it is not possible to change the key/cert file, unless you delete the email account in iOS and re-configure it.

(Even when we migrate to a new server, we keep /etc/exim.cert , /etc/exim.key as much as possible to keep end-user impact minimal)
 
I do empathize with your situation. It really sucks when stuff breaks out of the blue. I would be angry just as well.

It is a really bad luck that new apache version got released around the same time we made these changes. It made it look like it is new apache version problem ? .

The new release is being rolled out slowly, to give us a chance to make changes if needed. So far there are no changes that we could make to make this transition easier.

Adding an extra check to prevent file from being copied if it is empty is tempting, but I do not think we should not do it. Again it adds more complex and unexpected behaviour in a place where it should not. Instead of immediate broken apache you would get a really strange behaviour at some future point in time, when the empty file would be updated and syncing code would start replacing them (which previously was disabled because source files were empty). Or strange behaviour when some services that use separate files gets certificate synced, while other services that used combined cert file does not get synced.

Since this certificate installation by CustomBuild is a breaking change for anyone who expected it NOT to happen, there is little value in making it conditional. Failing fast makes it easier for everyone to understand why and how it failed and adjust to it.

Now looking at the bright side of this change - if service certificates were being managed manually it can be simplified by relying on CB to do it. By copying certs for DA and calling da build sync_server_cert if CB ever changes service configuration (for example cert location) it will make sure the certificates and service configuration in always in sync. This might mean you no longer need to customize apache configs to use different TLS certificate configuration (using separate chain and cert files).

Any ideas on how to make this transition easier are welcome. I hope this thread will be helpful in explaining how CB works starting DA 1.662.
 
ccto said:
(Even when we migrate to a new server, we keep /etc/exim.cert , /etc/exim.key as much as possible to keep end-user impact minimal)
Click to expand...

@ccto does it mean you do have servers where exim certificates are deliberately not in sync with DA or apache certificates?
 
fln said:
@ccto does it mean you do have servers where exim certificates are deliberately not in sync with DA or apache certificates?
Click to expand...

Yes, when iOS users setup mail account with that self signed cert in the past, it seems it stucks with that key/cert file.

If we change /etc/exim.key , /etc/exim.cert , they have no ways to connect to server (unless delete email account in iOS, and re-configure mail account in iOS)

So, we keep /etc/exim.cert , /etc/exim.key across each OS migration (unless those keys were too old, i.e. 1024 bit).

ref.: https://developer.apple.com/forums/thread/118658
 
Hmmm, then DA 1.662 will break things for you in a way we did not anticipate. When rebuilding exim or dovecot it will copy the certs over from directadmin conf directory. It is possible to make exim keys static by using custom cert/files and adding following lines to /etc/exim.variables.conf.custom:

Code: 
tls_certificate=/etc/exim.static.cert
tls_privatekey=/etc/exim.static.key

File paths can be anything except the default path. CB copies the certs to the fixed location where default config expects them to be.

For dovecot to use custom cert files custombuild/configure/dovecot/conf/ssl.conf would need to be copied to custombuild/custom/dovecot/conf/ssl.conf and updated to use other cert paths.
 
fln said:
Any ideas on how to make this transition easier are welcome. I hope this thread will be helpful in explaining how CB works starting DA 1.662.
Click to expand...
I'm feeling calmer now becuse I've built a workaround which solves this issue. But I still worry about the "next" breaking change around the corner.

If you're asking for ideas, I return to the one you rejected previously. Please put in an option variable, either for Custombuild or directadmin.conf.

I am a software developer, I built an ecommerce platform in 2005 that's still going strong today. During that time I've made a significant number of "breaking" changes to the software, but when I do this I make it a togglable option.

I typically set the feature off for existing customers and on by default for new customers. I then put out an update that informs all of my customers about the new feature, and how to turn it on. I illustrate the benefits of using it, but I know some won't.

This has worked well for years. DA used to do this, I have a number of settings in directadmin.conf that disable unwanted functionality. Yes, it does add bloat to the underlying code, but there's no significant performance impact. It's the cost of maintaining mature software, and my customers appreciate not having unpleasant surprises when they want to get on with selling their products.

This method works, trust me. Please consider it for changes such as this. Nobody wants to wake up to their Apache server down and their customers screaming at them. Even if the server admin is updating manually, it just takes too long to research the issue and find a solution under the pressure of their customers threatening to cancel their subscription and go elsewhere.
 
You must log in or register to reply here.
Back
Top