Apache Nginx / PHP FPM reload/restart issue

U

urbee

Verified User
Joined
Mar 27, 2009
Messages
118
Hello,

we have a strangely similar issue on Debian 7 than posted here:

https://forum.directadmin.com/archive/index.php/t-50718.html

We're running Debian 7, latest DA, primary php 5.5 fastcgi and secondary php-fpm 5.3, nginx_apache as webserver.

We're noticing random crashes of php-fpm where in logs we only see:

[05-Feb-2016 21:57:18] NOTICE: Reloading in progress ...
[05-Feb-2016 21:57:18] NOTICE: reloading: execvp("/usr/local/php53/sbin/php-fpm53", {"/usr/local/php53/sbin/php-fpm53", "--daemonize", "--pid=/var/run/php-fpm53.pid", "--fpm-config", "/usr/local/php53/etc/php-fpm.conf"})
[[05-Feb-2016 21:57:19] NOTICE: using inherited socket fd=142, "/usr/local/php53/sockets/user.sock"
[05-Feb-2016 21:57:19] NOTICE: fpm is running, pid 30863
[05-Feb-2016 21:57:19] NOTICE: ready to handle connections
[05-Feb-2016 21:57:20] NOTICE: Reloading in progress ...

After that, php-fpm goes out and we need to manually restart it for sites to start working again. This happens very randomly but mostly overnight, usually around 22:00 till 01:00 so i am not sure what this means. Also has happened during day but not as often.

As there are no errors anywhere i would be interested in someone taking a look for a fee ofcourse. smtalk? :)
 
Hmm.. hard to say what's causing that.
It might be a php bug, as the php-fpm bits are part of the php 5.3 compile.
The issue with that is php 5.3 has been EOL since August 2014, so they won't have any other updates available for it.

Aside from actually knowing why the reload randomly fails, we could narrow it down to confirm if it's a php-fpm issue.
Changing the php mode would be one way to do that, so perhaps try fastcgi for php 5.3 as well.

John
 
We're also seeing this behavior once in a couple of days. Haven't been able to track the issue down to it's core, but it seems to be only on PHP-FPM56. Not PHP-FPM70 (our second PHP version). We're running nginx_apache. Ubuntu 16.04.1 LTS, everything else works flawless by the way.

....
[24-Aug-2016 11:34:02] NOTICE: Reloading in progress ...
[24-Aug-2016 11:34:02] NOTICE: reloading: execvp("/usr/local/php56/sbin/php-fpm56", {"/usr/local/php56/sbin/php-fpm56", "--nodaemonize"})
[24-Aug-2016 11:34:03] NOTICE: fpm is running, pid 26054
[24-Aug-2016 11:34:03] NOTICE: ready to handle connections
[24-Aug-2016 11:34:03] NOTICE: systemd monitor interval set to 10000ms
[24-Aug-2016 11:36:02] NOTICE: Reloading in progress ...
[24-Aug-2016 11:36:02] NOTICE: reloading: execvp("/usr/local/php56/sbin/php-fpm56", {"/usr/local/php56/sbin/php-fpm56", "--nodaemonize"})
[24-Aug-2016 11:36:02] NOTICE: using inherited socket fd=7, "/usr/local/php56/sockets/{USERNAME}.sock"
{CUT, MANY LINES LIKE THE ONE ABOVE}
[24-Aug-2016 11:36:02] NOTICE: fpm is running, pid 26054
[24-Aug-2016 11:36:02] NOTICE: ready to handle connections
[24-Aug-2016 11:36:02] NOTICE: systemd monitor interval set to 10000ms
[24-Aug-2016 11:36:02] NOTICE: Reloading in progress ...
{MANUALLY RESTART PHP-FPM56}
[24-Aug-2016 11:48:53] NOTICE: Terminating ...
[24-Aug-2016 11:48:53] NOTICE: exiting, bye-bye!
.....
Click to expand...

Could it has anything to do with the fact that it was restarting at nearly the same second twice? (see the seconds).
 
The same issue just accorded again. We haven't had it for ~1-2 months or so. PHP 7.0 works fine.

[02-Dec-2016 15:55:01] NOTICE: Reloading in progress ...
[02-Dec-2016 15:55:01] NOTICE: reloading: execvp("/usr/local/php56/sbin/php-fpm56", {"/usr/local/php56/sbin/php-fpm56", "--nodaemonize"})
[02-Dec-2016 15:55:02] NOTICE: fpm is running, pid 22252
[02-Dec-2016 15:55:02] NOTICE: ready to handle connections
[02-Dec-2016 15:55:02] NOTICE: systemd monitor interval set to 10000ms
[02-Dec-2016 15:56:01] NOTICE: Reloading in progress ...
[02-Dec-2016 15:56:01] NOTICE: reloading: execvp("/usr/local/php56/sbin/php-fpm56", {"/usr/local/php56/sbin/php-fpm56", "--nodaemonize"})
[24-Aug-2016 11:36:02] NOTICE: using inherited socket fd=7, "/usr/local/php56/sockets/{USERNAME}.sock"
{CUT, MANY LINES LIKE THE ONE ABOVE}
[02-Dec-2016 15:56:02] NOTICE: fpm is running, pid 22252
[02-Dec-2016 15:56:02] NOTICE: ready to handle connections
[02-Dec-2016 15:56:02] NOTICE: systemd monitor interval set to 10000ms
[02-Dec-2016 15:56:02] NOTICE: Reloading in progress ...
[02-Dec-2016 16:06:01] NOTICE: Reloading in progress ...
[02-Dec-2016 16:06:02] NOTICE: Reloading in progress ...
[02-Dec-2016 16:07:01] NOTICE: Reloading in progress ...
[02-Dec-2016 16:07:02] NOTICE: Reloading in progress ...
{MANUALLY RESTART PHP-FPM56}
[02-Dec-2016 16:18:44] NOTICE: Terminating ...
[02-Dec-2016 16:18:44] NOTICE: exiting, bye-bye!
[02-Dec-2016 16:18:44] NOTICE: fpm is running, pid 31008
[02-Dec-2016 16:18:44] NOTICE: ready to handle connections
[02-Dec-2016 16:18:44] NOTICE: systemd monitor interval set to 10000ms
Click to expand...
 
Last edited:
We had this sort of also about 6 Months ore more ago after we did updates for php 5.6 and php7, we changed to only php 5.6 but problem still exists
Then after snapshot restore and update also problem exist with both php version, snapshot back only php 5.6 then reboot , updates and reboot with only php 5.6 worked. ( after this only using 1 php version per Server, we had more probs with PHP 7 not fully compatible for some webapps also as suhosin and co)

Also SMtalk had have a look a it and find a DA Custumbuildbug problem with systemd not in conf /starting somewhere ( after update Apache en rewrite confs) this was solved in DA CB 2 after that, don't know or you re using newest CB, and after which updates this happens.

The logs that i could find didn't say any usefull things.
 
I'm running latest versions of PHP 5, PHP 7, custombuild 2 and DirectAdmin on Ubuntu 16.04 LTS. I'm currently rebuilding PHP5.6, in hope this will fix it. But that's just a shot in the dark.
 
Also check apache and rewrite conf's we did a full reboot after this problem.

If parts that are not fully compatible remains somewhere because php7: suhosin or whatever remains i suspect here maybe a problem

(And systemd not loaded while not in "conf" we had also but that should be fixed with the newer CB's SMtalk found this DA CB bug that time )
Because of this reloading.... to check also:
[02-Dec-2016 15:56:02] NOTICE: systemd monitor interval set to 10000ms
[02-Dec-2016 15:56:02] NOTICE: Reloading in progress ...
Click to expand...
 
Last edited:
Yeah, all did this. Not seems to fixed it, unfortunately. Lately haven't seen this happening, fortunately (~1 week or so?).
 
Interestingly that I have just found a topic about this same problem I have got for about 1 year and cannot solve it until now.

After trying many ways to find the root cause, I can get to only that it is much likely caused by PHP-FPM (only version 5.6) and might be something about closing communication with Apache. (I have never used NginX on this system.) It seems when a PHP-FPM process does not close communication, the related Apache thread will not finish its task and still stay in W state (seen from server-status page), and also the new request will cause the PHP-FPM to spawn a new process for it, and so for the other new ones until one of them reach the limit configuration or the system is lack of memory to server. Everytime that the problem happens, I found that the amount children threads of Apache are at the limit number I set in Apache configuration.

I tried to track the request that trig the situation but find no specific one.

Still have to restart the process of PHP-FPM 5.6 when it is crashed every few days or a week (my server does not host high traffic website otherwise it would be crashed daily rather.)
 
You must log in or register to reply here.
Back
Top