Errors with php-fpm.

Richard G

Verified User
Joined
Jul 6, 2008
Messages
13,771
Location
Maastricht
I'm running in to this kid of errors and it's causing white screens on for example forum software, this is from the php-fpm73.log file:
Code:
[11-Jan-2021 20:16:21] WARNING: [pool klaas] child 30085 exited with code 70 after 0.008986 seconds from start
[11-Jan-2021 20:16:21] NOTICE: [pool klaas] child 30086 started
[11-Jan-2021 20:16:22] WARNING: [pool klaas] child 30086 exited with code 70 after 0.502812 seconds from start
[11-Jan-2021 20:16:22] NOTICE: [pool klaas] child 30087 started
[11-Jan-2021 20:16:29] WARNING: [pool satfun] child 30047 exited with code 70 after 32.714847 seconds from start
[11-Jan-2021 20:16:29] NOTICE: [pool satfun] child 30092 started
[11-Jan-2021 20:16:35] WARNING: [pool satfun] child 30092 exited with code 70 after 5.563168 seconds from start
[11-Jan-2021 20:16:35] NOTICE: [pool satfun] child 30100 started
[11-Jan-2021 20:16:35] WARNING: [pool satfun] child 30100 exited with code 70 after 0.041921 seconds from start

So all those exit code 70.

When doing a status request for php-fpm73 I got this:

Code:
Redirecting to /bin/systemctl status php-fpm73.service
● php-fpm73.service - The PHP FastCGI Process Manager
   Loaded: loaded (/etc/systemd/system/php-fpm73.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-01-10 04:33:54 CET; 1 day 15h ago
  Process: 24796 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 14555 (php-fpm73)
   Status: "Processes active: 1, idle: 1, Requests: 57083, slow: 0, Traffic: 0req/sec"
   CGroup: /system.slice/php-fpm73.service
           ├─14555 php-fpm: master process (/usr/local/php73/etc/php-fpm.conf)
           ├─18082 php-fpm: pool sslstream
           └─30211 php-fpm: pool satfun

Jan 10 04:34:10 server24.server010.nl systemd[1]: [/etc/systemd/system/php-fpm73.service:54] Unknown lvalue 'ProtectControlGroups' in sectio...ervice'
Jan 10 04:34:10 server24.server010.nl systemd[1]: [/etc/systemd/system/php-fpm73.service:57] Unknown lvalue 'RestrictRealtime' in section 'Service'
Jan 10 04:34:10 server24.server010.nl systemd[1]: [/etc/systemd/system/php-fpm73.service:41] Unknown lvalue 'ProtectKernelModules' in sectio...ervice'
Jan 10 04:34:10 server24.server010.nl systemd[1]: [/etc/systemd/system/php-fpm73.service:48] Unknown lvalue 'ProtectKernelTunables' in secti...ervice'
Jan 10 04:34:10 server24.server010.nl systemd[1]: [/etc/systemd/system/php-fpm73.service:54] Unknown lvalue 'ProtectControlGroups' in sectio...ervice'
Jan 10 04:34:10 server24.server010.nl systemd[1]: [/etc/systemd/system/php-fpm73.service:57] Unknown lvalue 'RestrictRealtime' in section 'Service'
Jan 10 04:35:02 server24.server010.nl systemd[1]: Reloading The PHP FastCGI Process Manager.
Jan 10 04:35:02 server24.server010.nl systemd[1]: Reloaded The PHP FastCGI Process Manager.
Jan 11 00:49:01 server24.server010.nl systemd[1]: Reloading The PHP FastCGI Process Manager.
Jan 11 00:49:01 server24.server010.nl systemd[1]: Reloaded The PHP FastCGI Process Manager.
Unknown value's? Seems wrong to me.

The white screen can be undone by waiting longer, or best by restarting php-fpm.

I have enough resources:
Code:
top - 20:21:21 up 16 days, 18:37,  1 user,  load average: 0.08, 0.18, 0.22
Tasks: 297 total,   1 running, 296 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 99.8 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 65714844 total, 32504800 free,  8625696 used, 24584348 buff/cache
KiB Swap: 33521660 total, 33521660 free,        0 used. 56404944 avail Mem

These white screens happen unregularly, what can be causing this?
 
Thank you my friend.
1st link, I did not use any override files anywhere.

2nd link, also php 7.4 but I don't see a solution, I've got more then enough memory free.

3rd link seems to point to the same result I'm having.

But I can't find anything which points to the blank screens or a solution.
 

Sounds like it memory related. Not that the memory you have is not enough but maybe not enough is assigned to the application.
 
At php fpm you can set memory size in pool max min auto and so, maybe that. I have it not in my head now but look at php fpm memory pool settings i gues.

Or other min max auto size for ..
I did it once longer time ago pool is for da user

Something with static , limits or so

Also wen a application or plugin has problems as looping for example.

Wen did this started do you know that?

It looks like you only have there two pools satfun and sslstream , sorry to long ago for me i was updated in my brains ... ;)


https://www.php.net/manual/en/install.fpm.configuration.php#125502 go to but only a tip how not to config
 
Last edited:
Yeah it might be memory assigned, but since I have memory enough it's odd. But I'm sorry. In this case I'd rather had some specific help then links to similar issues.
So I mean where do I have to look in Directadmin, what test or commands do I have to do to check?
Since it's not a structural problem but occurs once a while

How to check if this is really memory?

I have a default php-fpm with only in the directadmin.conf this change:
Code:
php_fpm_max_children_default=30

I presume the default memory setting used by DA would be enough, where is this configured so I can read and state the value?

I also don't have a clue on user specific php-fpm logs or where they would be in DA.

Edit: This is the php-fpm73.conf memory line:
Code:
;php_admin_value[memory_limit] = 32M
but it's commented by semicolumn so I presume it can use more.
 
This is rest:
Code:
pm = ondemand
pm.max_children = 10
pm.process_idle_timeout = 60
pm.max_requests = 1000

But the max children would be overriden by the 30 in the directadmin.conf.
 
This is rest:
Code:
pm = ondemand
pm.max_children = 10
pm.process_idle_timeout = 60
pm.max_requests = 1000

But the max children would be overriden by the 30 in the directadmin.conf.
Sorry i don't remember where id did changes and can't find the DA parts.
Was not on a DA box i did it!

Quick copy past of some notes but not DA related in host file or conf...

Question 2: In case it is no possible and if I have to override some php values from the Loaded Configuration I guess I have to edit the file /etc/php[version]/fpm/pool.d/[number].conf. Values format are like this "php_admin_value[allow_url_fopen] = 0". Can I insert these values in Virtualmin interface only for that virtual host?
https://httpd.apache.org/docs/2.4/mod/mod_proxy_fcgi.html
/etc/httpd/conf/httpd.conf that apply to this virtual server, directory or files


Examples​

Remember, in order to make the following examples work, you have to enable mod_proxy and mod_proxy_fcgi.

Single application instance​

ProxyPass "/myapp/" "fcgi://localhost:4000/"
mod_proxy_fcgi disables connection reuse by default, so after a request has been completed the connection will NOT be held open by that httpd child process and won't be reused. If the FastCGI application is able to handle concurrent connections from httpd, you can opt-in to connection reuse as shown in the following example:

Single application instance, connection reuse (2.4.11 and later)​

ProxyPass "/myapp/" "fcgi://localhost:4000/" enablereuse=on

Enable connection reuse to a FCGI backend like PHP-FPM​

Please keep in mind that PHP-FPM (at the time of writing, February 2018) uses a prefork model, namely each of its worker processes can handle one connection at the time.
By default mod_proxy (configured with enablereuse=on) allows a connection pool of ThreadsPerChild connections to the backend for each httpd process when using a threaded mpm (like worker or event), so the following use cases should be taken into account:
  • Under HTTP/1.1 load it will likely cause the creation of up to MaxRequestWorkers connections to the FCGI backend.
  • Under HTTP/2 load, due to how mod_http2 is implemented, there are additional h2 worker threads that may force the creation of other backend connections. The overall count of connections in the pools may raise to more than MaxRequestWorkers.
The maximum number of PHP-FPM worker processes needs to be configured wisely, since there is the chance that they will all end up "busy" handling idle persistent connections, without any room for new ones to be established, and the end user experience will be a pile of HTTP request timeouts.
The following example passes the request URI as a filesystem path for the PHP-FPM daemon to run. The request URL is implicitly added to the 2nd parameter. The hostname and port following fcgi:// are where PHP-FPM is listening. Connection pooling/reuse is enabled.

PHP-FPM​

ProxyPassMatch "^/myapp/.*\.php(/.*)?$" "fcgi://localhost:9000/var/www/" enablereuse=on
The following example passes the request URI as a filesystem path for the PHP-FPM daemon to run. In this case, PHP-FPM is listening on a unix domain socket (UDS). Requires 2.4.9 or later. With this syntax, the hostname and optional port following fcgi:// are ignored.

PHP-FPM with UDS​

ProxyPassMatch "^/(.*\.php(/.*)?)$" "unix:/var/run/php5-fpm.sock|fcgi://localhost/var/www/"
The balanced gateway needs mod_proxy_balancer and at least one load balancer algorithm module, such as mod_lbmethod_byrequests, in addition to the proxy modules listed above. mod_lbmethod_byrequests is the default, and will be used for this example configuration.

Balanced gateway to multiple application instances​

ProxyPass "/myapp/" "balancer://myappcluster/"
<Proxy "balancer://myappcluster/">
BalancerMember "fcgi://localhost:4000"
BalancerMember "fcgi://localhost:4001"
</Proxy>
You can also force a request to be handled as a reverse-proxy request, by creating a suitable Handler pass-through. The example configuration below will pass all requests for PHP scripts to the specified FastCGI server using reverse proxy. This feature is available in Apache HTTP Server 2.4.10 and later. For performance reasons, you will want to define a worker representing the same fcgi:// backend. The benefit of this form is that it allows the normal mapping of URI to filename to occur in the server, and the local filesystem result is passed to the backend. When FastCGI is configured this way, the server can calculate the most accurate PATH_INFO.

Proxy via Handler​

<FilesMatch "\.php$">
# Note: The only part that varies is /path/to/app.sock
SetHandler "proxy:unix:/path/to/app.sock|fcgi://localhost/"
</FilesMatch>

# Define a matching worker.
# The part that is matched to the SetHandler is the part that
# follows the pipe. If you need to distinguish, "localhost; can
# be anything unique.
<Proxy "fcgi://localhost/" enablereuse=on max=10>
</Proxy>

<FilesMatch ...>
SetHandler "proxy:fcgi://localhost:9000"
</FilesMatch>

<FilesMatch ...>
SetHandler "proxy:balancer://myappcluster/"
</FilesMatch>
top

Environment Variables​

In addition to the configuration directives that control the behaviour of mod_proxy, there are a number of environment variables that control the FCGI protocol provider:
proxy-fcgi-pathinfo When configured via ProxyPass or ProxyPassMatch, mod_proxy_fcgi will not set the PATH_INFO environment variable. This allows the backend FCGI server to correctly determine SCRIPT_NAME and Script-URI and be compliant with RFC 3875 section 3.3. If instead you need mod_proxy_fcgi to generate a "best guess" for PATH_INFO, set this env-var. This is a workaround for a bug in some FCGI implementations. This variable can be set to multiple values to tweak at how the best guess is chosen (In 2.4.11 and later only): first-dot PATH_INFO is split from the slash following the first "." in the URL. last-dot PATH_INFO is split from the slash following the last"." in the URL. full PATH_INFO is calculated by an attempt to map the URL to the local filesystem. unescape PATH_INFO is the path component of the URL, unescaped / decoded. any other value PATH_INFO is the same as the path component of the URL. Originally, this was the only proxy-fcgi-pathinfo option.
top

ProxyFCGIBackendType Directive​

Description:Specify the type of backend FastCGI application
Syntax:ProxyFCGIBackendType FPM|GENERIC
Default:ProxyFCGIBackendType FPM
Context:server config, virtual host, directory, .htaccess
Status:Extension
Module:mod_proxy_fcgi
Compatibility:Available in version 2.4.26 and later
This directive allows the type of backend FastCGI application to be specified. Some FastCGI servers, such as PHP-FPM, use historical quirks of environment variables to identify the type of proxy server being used. Set this directive to "GENERIC" if your non PHP-FPM application has trouble interpreting environment variables such as SCRIPT_FILENAME or PATH_TRANSLATED as set by the server.
One example of values that change based on the setting of this directive is SCRIPT_FILENAME. When using mod_proxy_fcgi historically, SCRIPT_FILENAME was prefixed with the string "proxy:fcgi://". This variable is what some generic FastCGI applications would read as their script input, but PHP-FPM would strip the prefix then remember it was talking to Apache. In 2.4.21 through 2.4.25, this prefix was automatically stripped by the server, breaking the ability of PHP-FPM to detect and interoperate with Apache in some scenarios.
top

ProxyFCGISetEnvIf Directive​

Description:Allow variables sent to FastCGI servers to be fixed up
Syntax:ProxyFCGISetEnvIf conditional-expression [!]environment-variable-name [value-expression]
Context:server config, virtual host, directory, .htaccess
Status:Extension
Module:mod_proxy_fcgi
Compatibility:Available in version 2.4.26 and later
Just before passing a request to the configured FastCGI server, the core of the web server sets a number of environment variables based on details of the current request. FastCGI programs often uses these environment variables as inputs that determine what underlying scripts they will process, or what output they directly produce.
Examples of noteworthy environment variables are:
  • SCRIPT_NAME
  • SCRIPT_FILENAME
  • REQUEST_URI
  • PATH_INFO
  • PATH_TRANSLATED
This directive allows the environment variables above, or any others of interest, to be overridden. This directive is evaluated after the initial values for these variables are set, so they can be used as input into both the condition expressions and value expressions.
Parameter syntax:
conditional-expression Specifies an expression that controls whether the environment variable that follows will be modified. For information on the expression syntax, see the examples that follow or the full specification at the ap_expr documentation. environment-variable-name Specifies the CGI environment variable to change, such as PATH_INFO. If preceded by an exclamation point, the variable will be unset. value-expression Specifies the replacement value for the preceding environment variable. Backreferences, such as "$1", can be included from regular expression captures in conditional-expression. If omitted, the variable is set (or overridden) to an empty string — but see the Note below. # A basic, unconditional override
ProxyFCGISetEnvIf "true" PATH_INFO "/example"

# Use an environment variable in the value
ProxyFCGISetEnvIf "true" PATH_INFO "%{reqenv:SCRIPT_NAME}"

# Use captures in the conditions and backreferences in the replacement
ProxyFCGISetEnvIf "reqenv('PATH_TRANSLATED') =~ m|(/.*prefix)(\d+)(.*)|" PATH_TRANSLATED "$1$3"

Note: Unset vs. Empty​

The following will unset VARIABLE, preventing it from being sent to the FastCGI server: ProxyFCGISetEnvIf true !VARIABLE
Whereas the following will erase any existing value of VARIABLE (by setting it to the empty string), but the empty VARIABLE will still be sent to the server: ProxyFCGISetEnvIf true VARIABLE
The CGI/1.1 specification does not distinguish between a variable with an empty value and a variable that does not exist. However, many CGI and FastCGI implementations distinguish (or allow scripts to distinguish) between the two. The choice of which to use is dependent upon your implementation and your reason for modifying the variable.
 
Last edited:
This is rest:
Code:
pm = ondemand
pm.max_children = 10
pm.process_idle_timeout = 60
pm.max_requests = 1000

But the max children would be overriden by the 30 in the directadmin.conf.
You have a section in DA GUI for manual changes per hostfile httpd, in one of those template parts mayby , but take care of backups and SSH acces let it open at those times you make changes!

I did some there with ifs for domainname or user. also sorry i am only remember some parts where id did change things for the php fpm pool and settings.

You can check a lot however ofcourse with an php.info file

Look here @Richard G https://help.directadmin.com/item.php?id=672 per domain maby helps

And here per user https://help.directadmin.com/item.php?id=636
 
Last edited:
I don't think I can use a per domain or user setting, because as you can see here, it happens already with 2 users.
However, you brought me on the right track.

I just doublechecked the php.ini and on the other servers it's at 512 MB so should be way enough.
However, on this server it was only 128 while the forums often need more. Then I remembered I put this back, to see if when the php ini settings were enabled, this could be adjusted per domain by the php settings, which was possible.
Then I set it back, but forgot to raise it again.

So I punt the memory limit now to 512 MB and see if the error will occur again.

It might be that this is the culprit, we have to wait out.

Also @bdacus01 thank you for your input, as always my friend, I just needed some place where to look. ;)
Hopefully this was it.
 
At this moment no hangers or whitescreens anymore, but still regularly the exit code 70.
Code:
[14-Jan-2021 10:52:38] WARNING: [pool mooirdam] child 22760 exited with code 70 after 0.009370 seconds from start
[14-Jan-2021 10:52:38] NOTICE: [pool mooirdam] child 22761 started
[14-Jan-2021 10:52:40] WARNING: [pool mooirdam] child 22761 exited with code 70 after 1.820305 seconds from start
[14-Jan-2021 10:52:40] NOTICE: [pool mooirdam] child 22763 started
and also some other accounts.
 
At this moment no hangers or whitescreens anymore, but still regularly the exit code 70.
Code:
[14-Jan-2021 10:52:38] WARNING: [pool mooirdam] child 22760 exited with code 70 after 0.009370 seconds from start
[14-Jan-2021 10:52:38] NOTICE: [pool mooirdam] child 22761 started
[14-Jan-2021 10:52:40] WARNING: [pool mooirdam] child 22761 exited with code 70 after 1.820305 seconds from start
[14-Jan-2021 10:52:40] NOTICE: [pool mooirdam] child 22763 started
and also some other accounts.
A "common" thing between account that have that error warnings!
( Same software / cms / updates at time / plugins) ?
 
( Same software / cms / updates at time / plugins) ?
Two of the accounts are running Wordpress if I'm not mistaken, the other one is running forum software, no WP at all.
So no common things that I know of.

The other server, same setup, also containt wp sites, not a single code 70.
 
The unknown values seem to come from an outdated systemd service (see this). So you can ignore it or update your systemd service.

Did you check if swapping happens? If not, it must be a memory limit of your PHP setup or webserver.

Do you reach your max. children limit? See this.
 
The unknown values seem to come from an outdated systemd service (see this).
 
The unknown values seem to come from an outdated systemd service
The errata points to Centos 7 and is closed. We are running Centos 8 on the servers so we have newer systemd's I guess.
However, after raising the php memory limit from 128 to 512 I did not see those unknown values anymore until now. But if I can ignore those unknown values, it's fine anyway. Not a big deal.

I was more worried about the white screens and php-fpm just seem to hang and cause white screens without anything to be seen in the log except for those error 70 exits.

I did read something in the the last post of your CP link about running opcache. We do run opcache. Is that a problem?
Which value do you use for max_children? Because that seems to me the only option left.
Max_children was already raised to 30 (default is 10). I could easily raise more but I thougth the default was already enough. It's just odd that the other server which has the exact same setup does not have this issue, not even with max_children at 10.
On check I don't even come near the 30. However, this might happen on some peaks. Last time of the error 70 was the day before yesterday. So I have to check again when I see the error 70 again in the logs.
Is that max_children per user pool, or in total? I thought per user, correct?

Swapping can be done, I do have an active swap partition, on /md0 but not sure if it's used.
Memory limit in php is set to 512M and we had never troubles before when using just mod_php with mod_ruid2. 512M must be more than enough.
And we have 64GB RAM with always over 30 GB free on a Intel Core i7-8700 CPU.

@bdacus01 Sorry, you pointed out that, but I was only looking for the error 70 mainly. However, I also thought it might be something else because the errata was closed 2 years ago for Centos 7 with the statement it would be fixed. And we're running Centos 8 so I thought something else was wrong.
 
I'm not sure about a way to check the children limit nor how to debug that properly beside analysing the logs (e. g. how often this occurs) and what the system load was, e. g. network traffic. It should match: high traffic, many exits. I read about having max_children set to 500 so I think you can just play around with this value. If it doesn't happen again, you know that it was the max_children issue.
 
I can check if too many children are running but only when the log contains the errors with this command:
Code:
ps aux | grep "php-fpm: pool"
which will show all running pools.

I read about having max_children set to 500
That high? Wow.

However, I just found in a feature of DA that the default was previously 100.
I thought this might haven been way to high, because of these posts in the statement in the feature (I made it bold):
We had previously increased the php-fpm setting up to 100 after a few reports of issues where it wasn't high enough. For the bulk of everyone using it, 10 should be plenty for 99.99% of all websites, and 100 is an edge case.
and
If you know you don't need this 100 limit (and you probably don't), then just add: php_fpm_max_children_default=10 to your directadmin.conf and restart directadmin.
So I thought by raising it to 30 I would already be a bit high. ;)


So it looks then if the 10 foor 99.99% does not look like a correct statement.
 
10 is for sure not a value that will serve 99.99 % on the market. I think having a too high rate won't be an issue. I mean, they are just used if they needed. So it's not much of trouble for a strong server like yours.
 
That reassures me a bit, and the fact that also some sites which explain calculation methods for using php-fpm settings also often have more max_children with a lot less memory.
So I can easily experiment with this. The statements really took me on the wrong track.

Thanks!
 
Back
Top