DirectAdmin Tally: Stuck

Erulezz

Verified User
Joined
Sep 14, 2015
Messages
905
Location
🇳🇱
I run the Tally command every 4 hours like described here;

https://help.directadmin.com/item.php?id=48

After that, PHP-FPM also reloads. I just had one system where PHP-FPM crashed and was returning 502 errors. It was solved with a simple restart php-fpm.
I didn't find any errors for users or in general except for this one thing in /var/log/php-fpm72.log;

Normal reload:

Code:
[23-Jul-2018 16:11:02] NOTICE: Reloading in progress ...
[23-Jul-2018 16:11:02] NOTICE: reloading: execvp("/usr/local/php72/sbin/php-fpm72", {"/usr/local/php72/sbin/php-fpm72", "--nodaemonize"})
[23-Jul-2018 16:11:02] NOTICE: using inherited socket fd=7
etc for more users
[23-Jul-2018 16:11:02] NOTICE: fpm is running, pid 18457
[23-Jul-2018 16:11:02] NOTICE: ready to handle connections
[23-Jul-2018 16:11:02] NOTICE: systemd monitor interval set to 10000ms

But now with this crash the only line I saw was

Code:
[23-Jul-2018 20:11:01] NOTICE: Reloading in progress ...

And it stops there. So for some vague reason the reloading is stuck and php-fpm is returning errors. I don't know where to debug this further or what exactly went wrong because this is the only "error" I could find. Is this a bug in the tally system or something else?
 
After days of investigating it turns out my server has the exact same problem. Although we run Tally only once a day (around midnight) the result is the same, php-fpm is not running sometimes and the php-fpm.log log shows the same reloading state:

Code:
[12-Feb-2019 10:12:02] NOTICE: Reloading in progress ...

If it works it shows something like this:

Code:
[11-Feb-2019 15:08:01] NOTICE: Reloading in progress ...
[11-Feb-2019 15:08:01] NOTICE: reloading: execvp("/usr/local/php72/sbin/php-fpm72", {"/usr/local/php72/sbin/php-fpm72", "--nodaemonize", "--pid=/run/php-fpm72.pid"})

Have you found a solution in the meantime?
 
No, I haven't. In all the years of using DirectAdmin this was the only time this has happened. I don't know if this is a bug in DirectAdmin or php-fpm..

Now that you also have this problem, it might be useful to create a ticket so that it can be investigated. Will do this later. It is quite a big problem if this happens ... Especially because php-fpm is "down", but the Service Monitor does not kick in and restart fpm because the php-fpm process runs technically but does not work.
 
We use monitoring tools to check whether or not /usr/local/php*/sockets/webapps.sock exists, if it does not, then PHP-FPM gets restarted.
 
It seems to be related to php-fpm only. It has happened to me when I manually restart the php-fpm service. Unfortunately it doesn't happen all the time, so it's really hard to investigate and fix this issue.

I'll see if I can monitor the webapps.sock file somehow, thanks for the tip.
 
Thanks! Ended up setting up Monit with the following check:

Code:
check file webapps.sock with path /usr/local/php72/sockets/webapps.sock
    if does not exist then exec "/bin/systemctl start php-fpm72.service"

It just started itself when I stopped it manually, so it seems to do the job!
 
Thanks, that looks very helpful!

@roeland;

Shouldn't it be restart instead of start? When this error happens php-fpm is technically running, and it needs a restart to fix this error.
 
We use restart with any action in monit even when we need to start a service which is not working.
 
Back
Top