Errors with php-fpm.

Richard G · Feb 4, 2021

Well... upping it again. Same issue again, and not even 100 childs in use by far. So I have the idea another user is causing this.
Maybe with a WP site. However, I can only see "php-fpm: pool klaas" but klaas has 15 domains. How can I see which domain is using the php-fpm pool of his account?
Because I would like to have a look in their apache files, but don't know which one of the 15 domains.

LawsHosting · Feb 5, 2021

Welcome to my world! (I know, useless reply).

By the way, my settings for just 1 site are:

Code:

pm = ondemand
pm.max_children = 10
pm.process_idle_timeout = 60
pm.max_requests = 1000

You tried raising max_requests?

Zhenyapan · Feb 5, 2021

Richard G said:
Well... upping it again. Same issue again, and not even 100 childs in use by far. So I have the idea another user is causing this.
Maybe with a WP site. However, I can only see "php-fpm: pool klaas" but klaas has 15 domains. How can I see which domain is using the php-fpm pool of his account?
Because I would like to have a look in their apache files, but don't know which one of the 15 domains.

check access/error logs for all domains of this account, you can parse only time when error in fpm log fixed.

Richard G · Feb 5, 2021

Peter Laws said:
Welcome to my world! (I know, useless reply).

No it's not useless. I've seen you also struggled with this, and every bit of possible solution might be of help.

I did not try raising max request, but for the whole server I do have max_children to 100 via directadmin.conf all the rest of the php-fpm config is default.
I presume I need to create custom php-fpm conf to be able to rais the max requests?

@Zhenyapan I was afraid of that. So I have to check 15-20 domain logs seperately. I'll have a try at that.

LawsHosting · Feb 5, 2021

Richard G said:
No it's not useless. I've seen you also struggled with this, and every bit of possible solution might be of help.

Yeah, copy to /usr/local/directadmin/data/templates/custom/php-fpm.conf then change |?MAX_REQUESTS=nnn|

It'll be good to have a MAX_REQUESTS option in directadmin.conf too.......

Richard G · Feb 5, 2021

I will try that. I just had the issue twice shortly after each other and really -nothing- is to be seen about it in the error log or normal domain log. Only normal lines.

Would indeed be better to have MAX_REQUESTS tool

Richard G · Feb 5, 2021

Just out of curiosity. Could a mysql/mariadb issue also cause php-fpm73 crashes with white screens?
Maybe my mariadb config is set too small or too little memory?

Code:

max_connections = 300
sql_mode=
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
tmp_table_size=512M
max_heap_table_size=512M
innodb_buffer_pool_size=6144M
innodb_buffer_pool_instances=4
innodb_log_file_size=256M
join_buffer_size=2M

P.s. max requests is already 1000 because that is the default, got this at the moment for php-fpm73:

Code:

pm = ondemand
pm.max_children = 100
pm.process_idle_timeout = 60
pm.max_requests = 1000

Zhenyapan · Feb 6, 2021

Mysql troubles can cause white screens, but not sure they can impact directly to PhpFpm.
In your config I see only one issue: you need to have one pool_instance per each pool_size GB,
so just modify strings to:
innodb_buffer_pool_size=6G
innodb_buffer_pool_instances=6
---
before you will do this - check mysqltuner recommendations.
----
also this may help: https://tideways.com/profiler/blog/an-introduction-to-php-fpm-tuning

Richard G · Feb 6, 2021

Thank you. I run mysqltuner some times, but not every time that gives recommendations that are good. In the past I did all it recommended and then I had troubles.
So I might need some help with that. I rather have that it's using too much resources then too little.
I use MariaDB and seems I don't have "performance schema=on" in my.cnf.

But this is the output:

Code:

General recommendations:
    /var/log/mysql/mysqld.log is > 32Mb, you should analyze why or implement a rotation log strategy such as logrotate!
    Control warning line(s) into /var/log/mysql/mysqld.log file
    Control error line(s) into /var/log/mysql/mysqld.log file
    Reduce or eliminate unclosed connections and network issues
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
             See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
             (specially the conclusions at the bottom of the page).
    Temporary table size is already large - reduce result set size
    Reduce your SELECT DISTINCT queries without LIMIT clauses
    Performance schema should be activated for better diagnostics
    Consider installing Sys schema from https://github.com/mysql/mysql-sys for MySQL
    Consider installing Sys schema from https://github.com/FromDual/mariadb-sys for MariaDB
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
    join_buffer_size (> 2.0M, or always use indexes with JOINs)
    table_definition_cache(400) > 3382 or -1 (autosizing if supported)
    performance_schema = ON enable PFS
    innodb_log_file_size should be (=768M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
    innodb_buffer_pool_instances(=6)

It's avery odd crash when it happens. Because php-fpm73 keeps running when I look to status, but not working anymore, nothing special in logs and only happens once a while, especially when I'm busy with that one forum for my customer, also Xenforo like here.
So that's why I think it might be related to MariaDB settings.

Other odd thing is that I -never- any issues like this with mod_php and mod_ruid2 with the same setup and config.

LawsHosting · Feb 6, 2021

Richard G said:
Other odd thing is that I -never- any issues like this with mod_php and mod_ruid2 with the same setup and config.

This..... Same with me, but I presume the webserver does everything in this setup, so configuring apache mpm is easier? Not sure. But, I agree, all this happened when I moved to php-fpm.

Richard G · Feb 6, 2021

There we go again, doing nothing special. Blank screen, php-fpm73 is just running, more then enough memory left and this is the output of the php-fpm73 status command.

Code:

● php-fpm73.service - The PHP FastCGI Process Manager
   Loaded: loaded (/etc/systemd/system/php-fpm73.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-02-07 01:30:02 CET; 27min ago
  Process: 19775 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 8512 (php-fpm73)
   Status: "Processes active: 1, idle: 1, Requests: 778, slow: 0, Traffic: 0.2req/sec"
   CGroup: /system.slice/php-fpm73.service
           ├─ 8512 php-fpm: master process (/usr/local/php73/etc/php-fpm.conf)          
           ├─20714 php-fpm: pool sslstream
           └─24831 php-fpm: pool satfun

Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:41] Unknown lvalue 'ProtectKernelModules' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:48] Unknown lvalue 'ProtectKernelTunables' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:51] Unknown lvalue 'ProtectControlGroups' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:54] Unknown lvalue 'RestrictRealtime' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:41] Unknown lvalue 'ProtectKernelModules' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:48] Unknown lvalue 'ProtectKernelTunables' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:51] Unknown lvalue 'ProtectControlGroups' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:54] Unknown lvalue 'RestrictRealtime' in section 'Service'
Feb 07 01:31:01 server.myserver.com systemd[1]: Reloading The PHP FastCGI Process Manager.
Feb 07 01:31:01 server.myserver.com systemd[1]: Reloaded The PHP FastCGI Process Manager.

So it says running cince 01:30:02 which indeed was after I had updated php.
Outage time 01.58 hours and according to this status, php-fpm73 is running, but every website is getting a white screen.

The only thing is some line about the forum software... have to ask about that.
Just discovered these issues started to happen after the forum software was installed, might be it's related.

Richard G · Feb 7, 2021

Zhenyapan said:
In your config I see only one issue: you need to have one pool_instance per each pool_size GB

I thought I looked that up when I put these lines in there so I went and doublechecked, but these values don't have an issue.

The docs say:

Each instance manages its own data structures and takes an equal portion of the total buffer pool size, so for example if innodb_buffer_pool_size is 4GB and innodb_buffer_pool_instances is set to 4, each instance will be 1GB. Each instance should ideally be at least 1GB in size.

So, in my case the pool sizes is set to 6 GB and instances are set to 4 which makes every instance 1,5G per piece, which is more then the minimum requirement of 1 GB.

Edit: Odd that MariaDB says "at least 1 GB" while mysqltuner advises to limit to 1 GB by advising 6 instances.

Zhenyapan · Feb 7, 2021

maybe my eyes lost "at least" from manuals when I read it

still getting 70 error?

ikkeben · Feb 7, 2021

Richard G said:

There we go again, doing nothing special. Blank screen, php-fpm73 is just running, more then enough memory left and this is the output of the php-fpm73 status command.

Code:

● php-fpm73.service - The PHP FastCGI Process Manager
   Loaded: loaded (/etc/systemd/system/php-fpm73.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-02-07 01:30:02 CET; 27min ago
  Process: 19775 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 8512 (php-fpm73)
   Status: "Processes active: 1, idle: 1, Requests: 778, slow: 0, Traffic: 0.2req/sec"
   CGroup: /system.slice/php-fpm73.service
           ├─ 8512 php-fpm: master process (/usr/local/php73/etc/php-fpm.conf)         
           ├─20714 php-fpm: pool sslstream
           └─24831 php-fpm: pool satfun

Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:41] Unknown lvalue 'ProtectKernelModules' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:48] Unknown lvalue 'ProtectKernelTunables' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:51] Unknown lvalue 'ProtectControlGroups' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:54] Unknown lvalue 'RestrictRealtime' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:41] Unknown lvalue 'ProtectKernelModules' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:48] Unknown lvalue 'ProtectKernelTunables' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:51] Unknown lvalue 'ProtectControlGroups' in section 'Service'
Feb 07 01:30:15 server.myserver.com systemd[1]: [/etc/systemd/system/php-fpm73.service:54] Unknown lvalue 'RestrictRealtime' in section 'Service'
Feb 07 01:31:01 server.myserver.com systemd[1]: Reloading The PHP FastCGI Process Manager.
Feb 07 01:31:01 server.myserver.com systemd[1]: Reloaded The PHP FastCGI Process Manager.

So it says running cince 01:30:02 which indeed was after I had updated php.
Outage time 01.58 hours and according to this status, php-fpm73 is running, but every website is getting a white screen.

The only thing is some line about the forum software... have to ask about that.
Just discovered these issues started to happen after the forum software was installed, might be it's related.

Don't know but check forum software and plugins compatible with your php version and config / settings , also mariadb version. ( mariadb yes or no strict and so on, but then i think other errors in log)

Richard G · Feb 7, 2021

I just editted my post with the last line. A bit odd isn't it, mysqltuner and the docs....

Yep, had some again, not causing white screens this time:

Code:

[07-Feb-2021 22:07:15] WARNING: [pool nultienfm] child 13296 exited with code 70 after 1.979574 seconds from start
[07-Feb-2021 22:07:15] NOTICE: [pool nultienfm] child 13301 started
[07-Feb-2021 22:07:15] WARNING: [pool nultienfm] child 13301 exited with code 70 after 0.286265 seconds from start
[07-Feb-2021 22:07:15] NOTICE: [pool nultienfm] child 13302 started
[07-Feb-2021 22:07:21] WARNING: [pool klaas] child 13308 exited with code 70 after 0.022316 seconds from start
[07-Feb-2021 22:07:21] NOTICE: [pool klaas] child 13309 started
[07-Feb-2021 22:07:22] WARNING: [pool klaas] child 13309 exited with code 70 after 0.214951 seconds from start
[07-Feb-2021 22:07:22] NOTICE: [pool klaas] child 13310 started
[07-Feb-2021 22:07:25] WARNING: [pool satfun] child 13314 exited with code 70 after 0.009040 seconds from start
[07-Feb-2021 22:07:25] NOTICE: [pool satfun] child 13315 started
[07-Feb-2021 22:07:29] WARNING: [pool satfun] child 13315 exited with code 70 after 4.178245 seconds from start
[07-Feb-2021 22:07:29] NOTICE: [pool satfun] child 13323 started
[07-Feb-2021 22:07:34] WARNING: [pool satfun] child 13323 exited with code 70 after 4.916632 seconds from start
[07-Feb-2021 22:07:34] NOTICE: [pool satfun] child 13331 started
[07-Feb-2021 22:07:37] WARNING: [pool uituwradio] child 13338 exited with code 70 after 0.022146 seconds from start
[07-Feb-2021 22:07:37] NOTICE: [pool uituwradio] child 13339 started
[07-Feb-2021 22:07:39] WARNING: [pool sslstream] child 9912 exited on signal 9 (SIGKILL) after 673.979610 seconds from start
[07-Feb-2021 22:07:39] NOTICE: [pool sslstream] child 13344 started
[07-Feb-2021 22:11:01] NOTICE: Reloading in progress ...
[07-Feb-2021 22:11:01] NOTICE: reloading: execvp("/usr/local/php73/sbin/php-fpm73", {"/usr/local/php73/sbin/php-fpm73", "--nodaemonize", "--pid=/run/php-fpm73.pid"})

I don't know if that last is an automatic reload.

Once a while I see such reload (no errors and no warnings before it, just running fine)

Code:

[07-Feb-2021 22:19:01] NOTICE: using inherited socket fd=50, "/usr/local/php73/sockets/someuser.sock"
[07-Feb-2021 22:19:01] NOTICE: using inherited socket fd=50, "/usr/local/php73/sockets/someuser.sock"
[07-Feb-2021 22:19:01] NOTICE: fpm is running, pid 24902
[07-Feb-2021 22:19:01] NOTICE: ready to handle connections
[07-Feb-2021 22:19:01] NOTICE: systemd monitor interval set to 10000ms
[07-Feb-2021 22:20:01] NOTICE: Reloading in progress ...
[07-Feb-2021 22:20:01] NOTICE: reloading: execvp("/usr/local/php73/sbin/php-fpm73", {"/usr/local/php73/sbin/php-fpm73", "--nodaemonize", "--pid=/run/p
hp-fpm73.pid"})

I'm wondering about something else. Could there be some kind of confict? On the same server, we've got MediaCP running, which runs their own Nginx webstuff (we only use Apache 2.4 for hosting accounts) and also uses seperate php-fpm process.

Code:

root      1825  0.0  0.0 329536  6520 ?        Ss   Feb05   0:09 php-fpm: master process (/usr/local/mediacp/php/etc/php-fpm.conf)
mediacp   1826  0.0  0.0 407144 19396 ?        S    Feb05   0:00  \_ php-fpm: pool www
mediacp   1827  0.0  0.0 407148 17900 ?        S    Feb05   0:00  \_ php-fpm: pool www

Could this be causing some conflict which could cause this issue?

@ikkeben That is already checked before, that is same software as here and normally runs fine on 7.3 and 7.2 is even the minimal requirement.
I'm the only one with this issue and I also see error 70 from one or two other domains so might still be cause by something else.

LawsHosting · Feb 8, 2021

Is PHP actually crashing? I had something similar, and it turned out PHP wasn't compiling properly....... so a ./build clean;./build update;./build update_data, then rebuilding PHP(s) was needed.....

Richard G · Feb 8, 2021

Peter Laws said:
Is PHP actually crashing?

No. As you can see from the status command it is still running, but it's not parsing anymore on any php website.
A couple of days ago php was rebuild again because of php updates.

warg · Feb 8, 2021

Is it feasible for you to try PHP 8? Does `make test` show any issues?

ikkeben · Feb 8, 2021

Somewhere i have seen a change log / bug about DA and php -fpm child ...

Don't remember but was short time ago.

http://files.directadmin.com/services/custombuild/patches/fpm_children_cgroup.patch

Richard G · Feb 8, 2021

@warg No I might switch for 1 forum to php 7.4 we will certainly not implement php 8 yet.
I could do a "make test" for php 7.3 if that would not disturb any live site. But don't know where I can do that.
However during the compile this week when updating to 7.3.27 I can't remember seeing any difficulties or errors. Would be odd if this would happen only on 1 server and all others working fine.

I'm still thinking about a conflict with the php-fpm of the Mediacp too.

@ikkeben I checked for that file and it's present on the server.
Let's wait and see if it happens again. However if this would be the issue I wonder why I only encountered it on one server.
And I still had some exit code 70 on februari 7th which is 2 days after the data of this file on the server.

Errors with php-fpm.

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User