apache restarting randomly on all directadmin servers

[Driesp]

Hello

I noticed since we upgraded to DirectAdmin 1.697
that DirectAdmin is restarting the httpd (apache) service from time to time without a clear reason.

It is restarting with this message:
/var/log/messages:Mar 21 17:19:42 web1 dataskq[2852283]: executing task task=action=httpd&affect_php_fpm=no&value=restart
/var/log/messages:Mar 21 17:19:47 web1 dataskq[2852283]: finished task duration=4.942548506s task=action=httpd&affect_php_fpm=no&value=restart

It's not really a problem that it is restarting apache, it is a problem that restarting apache can take up to 10 seconds.
During this time, the webserver is offline, and it is rejecting new connections.

Before the update, DirectAdmin would never restart the apache webserver, it would request a reload if that was needed.

Is it a bug?

I will upgrade to DirectAdmin 1.698 to see if the problem persists.

Kind regards
Dries

 

[Driesp]

Update:

I don't know if this is relevant:
# /usr/local/directadmin/directadmin c | grep -E 'graceful_restarts|restart_apache_after_tally'
restart_apache_after_tally=1

While I have the option restart_apache_after_tally not set in directadmin.conf
# cat /usr/local/directadmin/conf/directadmin.conf | grep -E 'graceful_restarts|restart_apache_after_tally'
graceful_restarts=1


Kr
Dries

 

[Driesp]

Update

# /usr/local/directadmin/directadmin config-set graceful_restarts 1
Error with the current values: Cannot find 'graceful_restarts' in the directadmin.conf

# /usr/local/directadmin/directadmin config-set restart_apache_after_tally 0

# cat directadmin.conf | grep restart
graceful_restarts=1
restart_apache_after_tally=0

Kr
Dries

 

[Driesp]

I just noticed that 'graceful_restarts' has been removed since 1.654
Then I don't understand why it is restarting the service and not reloading it as it was before.

Kr
Dries

 

[J89]

We have the same on all our servers, it looks related with de SSL letsencrypt renew on every domain.

Normally it reloads apache only without seconds downtime, sinds a few weeks restart apache complete.

Hopefully they can change it back, it works really good before.

 

[Ohm J]

because that option is called "restart", but normally it's always reload the webserver after tally, isn't yet ?

so it might conflic with graceful_restarts, try to not manual set that option and re-test again.

 

[Driesp]

We have the same on all our servers, it looks related with de SSL letsencrypt renew on every domain.

Normally it reloads apache only without seconds downtime, sinds a few weeks restart apache complete.

Hopefully they can change it back, it works really good before.

It looks indeed like its the certificate renewal that is triggering a restart instead of a reload.
I am running the latest version of DirectAdmin now.

Mar 30 16:58:26  dataskq[2160877]: finished task             duration=5.637009126s task=action=rewrite&value=letsencrypt
Mar 30 16:58:26  dataskq[2160877]: executing task            task=action=service-watchdog
Mar 30 16:58:26  dataskq[2160877]: finished task             duration=23.6108ms task=action=service-watchdog

Mar 30 16:59:21  dataskq[2161195]: executing task            task=action=httpd&affect_php_fpm=no&value=restart
Mar 30 16:59:21  systemd[1]: Stopping The Apache HTTP Server...
Mar 30 16:59:26  systemd[1]: httpd.service: State 'stop-sigterm' timed out. Killing.
Mar 30 16:59:26  systemd[1]: httpd.service: Killing process 1068468 (httpd) with signal SIGKILL.
Mar 30 16:59:26  systemd[1]: httpd.service: Killing process 2157641 (httpd) with signal SIGKILL.
Mar 30 16:59:26  systemd[1]: httpd.service: Main process exited, code=killed, status=9/KILL
Mar 30 16:59:26  systemd[1]: httpd.service: Failed with result 'timeout'.
Mar 30 16:59:26  systemd[1]: Stopped The Apache HTTP Server.
Mar 30 16:59:26  systemd[1]: Starting The Apache HTTP Server...

It is not very nice to do a restart, as you can see, it is even killing processes.
What can we do about this?

Kr
Dries