graceful_restarts=1 not being fully honored on log rotation

Willis

Verified User
Joined
Dec 31, 2005
Messages
39
CentOS 7, DA 1.57.1, apache 2.4.39

I have a reverse proxy set up on Apache that proxies WebSockets. I am trying to keep them alive through the nightly tally process, but so far am not having any luck.

I followed the guide here: https://www.directadmin.com/features.php?id=980

It appears apache still gets a hard SIGHUP first to rotate logs (which disconnects all open sockets), then runs the tallies, then does a graceful restart when tallying is complete.

Here is the relevant apache error_log output I'm seeing:
[Sat Jun 22 00:10:02.824483 2019] [mpm_prefork:notice] [pid 21278] AH00173: SIGHUP received. Attempting to restart
[Sat Jun 22 00:10:03.254893 2019] [mpm_prefork:notice] [pid 21278] AH00163: Apache/2.4.39 (Unix) OpenSSL/1.0.2k-fips PHP/7.3.6 configured -- resuming normal operations
[Sat Jun 22 00:10:03.254934 2019] [core:notice] [pid 21278] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Sat Jun 22 00:11:01.323006 2019] [mpm_prefork:notice] [pid 21278] AH00171: Graceful restart requested, doing restart
[Sat Jun 22 00:11:01.527511 2019] [mpm_prefork:notice] [pid 21278] AH00163: Apache/2.4.39 (Unix) OpenSSL/1.0.2k-fips PHP/7.3.6 configured -- resuming normal operations
[Sat Jun 22 00:11:01.527535 2019] [core:notice] [pid 21278] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

DA system log looks like this:
2019:06:22-00:10:02: Tally All Begin
2019:06:22-00:10:02: Tally Reseller admin Begin
2019:06:22-00:10:02: Tally User user1 Begin
2019:06:22-00:10:16: Tally User user1 Complete
.... continued for each user
2019:06:22-00:10:40: Tally Reseller admin Complete
2019:06:22-00:10:40: Tally All Complete
2019:06:22-00:11:01: httpd reloaded

I have confirmed that the sockets do survive through a graceful restart/reload, by issuing a reload for Service Monitor, and the 00:11:xx graceful restart also does not disconnect any socket opened after the 00:10:xx restart.

Is there a way to make the 00:10:02 (apparently from log rotation) also graceful?

Edit 1:
After a bit of research, I've found that logrotate is issuing a full HUP via /etc/logrotate.d/apache and apache.logrotate/httpd_2 in custombuild seems to match and not take graceful_restarts into effect. I have modified /etc/logrotate.d/apache to call systemctl reload httpd.service instead of kill -HUP and can confirm at least doing that prevents a disconnect on apache's core log file rotation. Waiting until the next natural rotation to see if that works for when it rotates directadmin user apache logs as well.

Edit 2:
Adding graceful reload to logrotate for server-level apache logs was not enough. Whatever DirectAdmin is using to rotate all the apache logs for user-level domains is forcing a full SIGHUP which ignores graceful_restarts before it tallies statistics.
 
Last edited:
Bump, anybody have any useful insight here?

https://www.directadmin.com/features.php?id=507 would disable rotation completely, which is not what I want.
https://www.directadmin.com/features.php?id=980 & https://www.directadmin.com/features.php?id=1020 only fixes the post-tally (good, but only half the problem).
https://www.directadmin.com/features.php?id=926 doesn't allow anything to change the rotation hup (only force it off).

I haven't been able to locate a script/hook that would allow me to change the initial log rotation hup to be graceful.

I want logs to rotate daily, but I don't mind slight inaccuracies from long running sessions being on the wrong day throwing tallies off (or rather just delaying).
 
I may have found the culprit...

letsencrypt.sh inclues the line:
echo 'action=httpd&value=restart&affect_php_fpm=no' >> ${TASK_QUEUE}

https://www.directadmin.com/features.php?id=980 states dataskq should replace that with value=graceful, but it does not appear to be doing so. Is that check looking exactly for 'action=httpd&value=restart' and not matching if there are additional parameters?

I am modifying that to be value=graceful and I will see if the issue exists in the next tally action.

Edit:
forced a tally with echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
and unfortunately, this did not appear to resolve the issue, however echo 'action=httpd&value=restart&affect_php_fpm=no' >> /usr/local/directadmin/data/task.queue also does seem to force a full restart instead of graceful as well.

Edit 2:
echo 'action=httpd&value=restart&affect_php_fpm=no' >> /usr/local/directadmin/data/task.queue results in a non-graceful restart
echo 'action=httpd&value=graceful&affect_php_fpm=no' >> /usr/local/directadmin/data/task.queue results in a graceful restart/reload
however, modifying letsencrypt.sh to use value=graceful did not appear to work on tally. is DirectAdmin not using that script and handling letsencrypt internally?
 
Last edited:
So after testing a few more things, the general consensus I'm seeing is that when graceful_restarts=1 is set, it is only replacing the exact string of 'action=httpd&value=restart' and if there are additional parameters on it (such as 'action=httpd&value=restart&affect_php_fpm=no'), value is not changed to graceful.

Edit: it's not even doing a replace for 'action=httpd&value=restart' in the queue like KB980 states.
 
Last edited:
Hello,

For an assistance I would suggest opening a ticket with DirectAdmin support. It might speed up the process.
 
Back
Top