Unable to serve *any* page on fresh Apache httpd 2.4.41 - process aborts

Willis

Verified User
Joined
Dec 31, 2005
Messages
30
Fresh DA install on CentOS 8 (thought about posting in that forum, but the only issue I'm having is with httpd).

No installed users yet, just trying to view http://[ip] to see the default page, and it fails.

Code:
[Fri Oct 25 18:27:01.370075 2019] [ssl:warn] [pid 31011:tid 139754341652736] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
[Fri Oct 25 18:27:01.370236 2019] [suexec:notice] [pid 31011:tid 139754341652736] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Oct 25 18:27:01.382712 2019] [ssl:warn] [pid 31011:tid 139754341652736] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
[Fri Oct 25 18:27:01.383014 2019] [lbmethod_heartbeat:notice] [pid 31011:tid 139754341652736] AH02282: No slotmem from mod_heartmonitor
[Fri Oct 25 18:27:01.383034 2019] [:notice] [pid 31011:tid 139754341652736] mod_ruid2/0.9.8 enabled
[Fri Oct 25 18:27:01.384659 2019] [mpm_event:notice] [pid 31011:tid 139754341652736] AH00489: Apache/2.4.41 (Unix) OpenSSL/1.1.1 configured -- resuming normal operations
[Fri Oct 25 18:27:01.384693 2019] [core:notice] [pid 31011:tid 139754341652736] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Fri Oct 25 18:29:47.560338 2019] [core:notice] [pid 31011:tid 139754341652736] AH00052: child pid 31013 exit signal Aborted (6)
[Fri Oct 25 18:29:47.560382 2019] [core:notice] [pid 31011:tid 139754341652736] AH00052: child pid 31015 exit signal Aborted (6)
[Fri Oct 25 18:29:49.562700 2019] [core:notice] [pid 31011:tid 139754341652736] AH00052: child pid 32083 exit signal Aborted (6)
[Fri Oct 25 18:29:51.565157 2019] [core:notice] [pid 31011:tid 139754341652736] AH00052: child pid 32247 exit signal Aborted (6)
[Fri Oct 25 18:29:53.567755 2019] [core:notice] [pid 31011:tid 139754341652736] AH00052: child pid 32411 exit signal Aborted (6)
[Fri Oct 25 18:29:55.570264 2019] [core:notice] [pid 31011:tid 139754341652736] AH00052: child pid 32575 exit signal Aborted (6)
[Fri Oct 25 18:29:57.572635 2019] [core:notice] [pid 31011:tid 139754341652736] AH00052: child pid 32747 exit signal Aborted (6)
[Fri Oct 25 18:29:59.574257 2019] [core:notice] [pid 31011:tid 139754341652736] AH00052: child pid 451 exit signal Aborted (6)
[Fri Oct 25 18:30:01.576808 2019] [core:notice] [pid 31011:tid 139754341652736] AH00052: child pid 629 exit signal Aborted (6)
[Fri Oct 25 18:30:03.579142 2019] [core:notice] [pid 31011:tid 139754341652736] AH00052: child pid 869 exit signal Aborted (6)
All 10 exit signal Aborted's are triggered from a single http request of /.

I tried ratcheting the LogLevel up to debug and trace6, and nothing seems out of the ordinary aside from the AH00052 errors I am getting at the default warn log level.

Configured with php-fpm/php 7.2.24, but the errors are occuring on simple static html files, not even attempting to utilize php.

apachectl -M:

Code:
Loaded Modules:
 core_module (static)
 authn_file_module (static)
 authn_dbm_module (static)
 authn_anon_module (static)
 authn_dbd_module (static)
 authn_socache_module (static)
 authn_core_module (static)
 authz_host_module (static)
 authz_groupfile_module (static)
 authz_user_module (static)
 authz_dbm_module (static)
 authz_owner_module (static)
 authz_dbd_module (static)
 authz_core_module (static)
 access_compat_module (static)
 auth_basic_module (static)
 auth_form_module (static)
 auth_digest_module (static)
 allowmethods_module (static)
 file_cache_module (static)
 cache_module (static)
 cache_disk_module (static)
 cache_socache_module (static)
 socache_shmcb_module (static)
 socache_dbm_module (static)
 socache_memcache_module (static)
 socache_redis_module (static)
 so_module (static)
 watchdog_module (static)
 macro_module (static)
 dbd_module (static)
 dumpio_module (static)
 buffer_module (static)
 ratelimit_module (static)
 reqtimeout_module (static)
 ext_filter_module (static)
 request_module (static)
 include_module (static)
 filter_module (static)
 substitute_module (static)
 sed_module (static)
 deflate_module (static)
 http_module (static)
 mime_module (static)
 log_config_module (static)
 log_debug_module (static)
 logio_module (static)
 env_module (static)
 expires_module (static)
 headers_module (static)
 unique_id_module (static)
 setenvif_module (static)
 version_module (static)
 remoteip_module (static)
 proxy_module (static)
 proxy_connect_module (static)
 proxy_ftp_module (static)
 proxy_http_module (static)
 proxy_fcgi_module (static)
 proxy_scgi_module (static)
 proxy_uwsgi_module (static)
 proxy_fdpass_module (static)
 proxy_wstunnel_module (static)
 proxy_ajp_module (static)
 proxy_balancer_module (static)
 proxy_express_module (static)
 proxy_hcheck_module (static)
 session_module (static)
 session_cookie_module (static)
 session_dbd_module (static)
 slotmem_shm_module (static)
 ssl_module (static)
 http2_module (static)
 lbmethod_byrequests_module (static)
 lbmethod_bytraffic_module (static)
 lbmethod_bybusyness_module (static)
 lbmethod_heartbeat_module (static)
 unixd_module (static)
 dav_module (static)
 status_module (static)
 autoindex_module (static)
 info_module (static)
 suexec_module (static)
 cgi_module (static)
 dav_fs_module (static)
 dav_lock_module (static)
 vhost_alias_module (static)
 negotiation_module (static)
 dir_module (static)
 actions_module (static)
 speling_module (static)
 userdir_module (static)
 alias_module (static)
 rewrite_module (static)
 ruid2_module (shared)
 systemd_module (shared)
 mpm_event_module (shared)
I thought this might be a fluke, but this is two new CentOS 8 VMs from scratch, and they both had this issue out of the box. Installed from CentOS 8 netinstall on KVM.
Nothing I can identify as relevant hitting system message log.

Here is the full error log from trace6 modified to mask IP and removing redundant lines for spinning up http2:trace2 worker threads:

Code:
[Fri Oct 25 17:14:37.457263 2019] [mpm_event:trace6] [pid 10650:tid 140066810627840] event.c(1632): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Fri Oct 25 17:14:37.457358 2019] [http2:trace1] [pid 10650:tid 140068394911488] h2_h2.c(588): [client 256.256.256.256:19482] h2_h2, process_conn
[Fri Oct 25 17:14:37.457369 2019] [http2:trace1] [pid 10650:tid 140068394911488] h2_h2.c(602): [client 256.256.256.256:19482] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Fri Oct 25 17:14:37.457398 2019] [http2:trace2] [pid 10650:tid 140068394911488] h2_h2.c(639): [client 256.256.256.256:19482] h2_h2, not detected in 24 bytes(base64): R0VUIC8gSFRUUC8xLjENCkhvc3Q6IDEw
[Fri Oct 25 17:14:37.457407 2019] [http2:trace1] [pid 10650:tid 140068394911488] h2_h2.c(661): [client 256.256.256.256:19482] h2_h2, declined
[Fri Oct 25 17:14:37.457438 2019] [core:trace5] [pid 10650:tid 140068394911488] protocol.c(710): [client 256.256.256.256:19482] Request received from client: GET / HTTP/1.1
[Fri Oct 25 17:14:38.249615 2019] [core:notice] [pid 8293:tid 140069324589312] AH00052: child pid 10650 exit signal Aborted (6)
[Fri Oct 25 17:14:38.249658 2019] [core:trace4] [pid 8293:tid 140069324589312] mpm_common.c(536): mpm child 10650 (gen 0/slot 0) exited
[Fri Oct 25 17:14:39.250698 2019] [mpm_event:trace5] [pid 8293:tid 140069324589312] event.c(2886): Spawning new child: slot 0 active / total daemons: 0/0
[Fri Oct 25 17:14:39.250945 2019] [core:trace4] [pid 8293:tid 140069324589312] mpm_common.c(536): mpm child 10908 (gen 0/slot 0) started
[Fri Oct 25 17:14:39.252426 2019] [watchdog:debug] [pid 10908:tid 140069324589312] mod_watchdog.c(590): AH02981: Watchdog: Created child worker thread (_proxy_hcheck_).
[Fri Oct 25 17:14:39.252491 2019] [proxy:debug] [pid 10908:tid 140069324589312] proxy_util.c(1935): AH00925: initializing worker proxy:reverse shared
[Fri Oct 25 17:14:39.252506 2019] [proxy:debug] [pid 10908:tid 140069324589312] proxy_util.c(1992): AH00927: initializing worker proxy:reverse local
[Fri Oct 25 17:14:39.252525 2019] [proxy:debug] [pid 10908:tid 140069324589312] proxy_util.c(2027): AH00930: initialized pool in child 10908 for (*) min=0 max=159 smax=159
[Fri Oct 25 17:14:39.252554 2019] [http2:trace3] [pid 10908:tid 140069324589312] h2_conn.c(135): h2_workers: min=64 max=96, mthrpchild=64, idle_secs=600
[Fri Oct 25 17:14:39.254475 2019] [mpm_event:debug] [pid 10908:tid 140068403304192] event.c(2315): AH02471: start_threads: Using epoll (wakeable)
[Fri Oct 25 17:14:39.256282 2019] [http2:trace1] [pid 10908:tid 140068394911488] h2_h2.c(588): [client 256.256.256.256:13494] h2_h2, process_conn
[Fri Oct 25 17:14:39.256306 2019] [http2:trace1] [pid 10908:tid 140068394911488] h2_h2.c(602): [client 256.256.256.256:13494] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Fri Oct 25 17:14:39.256332 2019] [http2:trace2] [pid 10908:tid 140068394911488] h2_h2.c(639): [client 256.256.256.256:13494] h2_h2, not detected in 24 bytes(base64): R0VUIC8gSFRUUC8xLjENCkhvc3Q6IDEw
[Fri Oct 25 17:14:39.256338 2019] [http2:trace1] [pid 10908:tid 140068394911488] h2_h2.c(661): [client 256.256.256.256:13494] h2_h2, declined
[Fri Oct 25 17:14:39.256361 2019] [core:trace5] [pid 10908:tid 140068394911488] protocol.c(710): [client 256.256.256.256:13494] Request received from client: GET / HTTP/1.1
[Fri Oct 25 17:14:40.252152 2019] [core:notice] [pid 8293:tid 140069324589312] AH00052: child pid 10908 exit signal Aborted (6)
[Fri Oct 25 17:14:40.252197 2019] [core:trace4] [pid 8293:tid 140069324589312] mpm_common.c(536): mpm child 10908 (gen 0/slot 0) exited
[Fri Oct 25 17:14:41.253253 2019] [mpm_event:trace5] [pid 8293:tid 140069324589312] event.c(2886): Spawning new child: slot 0 active / total daemons: 0/0
[Fri Oct 25 17:14:41.253500 2019] [core:trace4] [pid 8293:tid 140069324589312] mpm_common.c(536): mpm child 11072 (gen 0/slot 0) started
[Fri Oct 25 17:14:41.255721 2019] [watchdog:debug] [pid 11072:tid 140069324589312] mod_watchdog.c(590): AH02981: Watchdog: Created child worker thread (_proxy_hcheck_).
[Fri Oct 25 17:14:41.255817 2019] [proxy:debug] [pid 11072:tid 140069324589312] proxy_util.c(1935): AH00925: initializing worker proxy:reverse shared
[Fri Oct 25 17:14:41.255836 2019] [proxy:debug] [pid 11072:tid 140069324589312] proxy_util.c(1992): AH00927: initializing worker proxy:reverse local
[Fri Oct 25 17:14:41.255858 2019] [proxy:debug] [pid 11072:tid 140069324589312] proxy_util.c(2027): AH00930: initialized pool in child 11072 for (*) min=0 max=159 smax=159
[Fri Oct 25 17:14:41.255888 2019] [http2:trace3] [pid 11072:tid 140069324589312] h2_conn.c(135): h2_workers: min=64 max=96, mthrpchild=64, idle_secs=600
[Fri Oct 25 17:14:41.259163 2019] [mpm_event:debug] [pid 11072:tid 140068403304192] event.c(2315): AH02471: start_threads: Using epoll (wakeable)
[Fri Oct 25 17:14:41.261259 2019] [http2:trace1] [pid 11072:tid 140068394911488] h2_h2.c(588): [client 256.256.256.256:34582] h2_h2, process_conn
[Fri Oct 25 17:14:41.261288 2019] [http2:trace1] [pid 11072:tid 140068394911488] h2_h2.c(602): [client 256.256.256.256:34582] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Fri Oct 25 17:14:41.261321 2019] [http2:trace2] [pid 11072:tid 140068394911488] h2_h2.c(639): [client 256.256.256.256:34582] h2_h2, not detected in 24 bytes(base64): R0VUIC8gSFRUUC8xLjENCkhvc3Q6IDEw
[Fri Oct 25 17:14:41.261331 2019] [http2:trace1] [pid 11072:tid 140068394911488] h2_h2.c(661): [client 256.256.256.256:34582] h2_h2, declined
[Fri Oct 25 17:14:41.261477 2019] [core:trace5] [pid 11072:tid 140068394911488] protocol.c(710): [client 256.256.256.256:34582] Request received from client: GET / HTTP/1.1
[Fri Oct 25 17:14:42.254606 2019] [core:notice] [pid 8293:tid 140069324589312] AH00052: child pid 11072 exit signal Aborted (6)
[Fri Oct 25 17:14:42.254666 2019] [core:trace4] [pid 8293:tid 140069324589312] mpm_common.c(536): mpm child 11072 (gen 0/slot 0) exited
Any thoughts or suggestions would be appreciated.

Edit:
Note, httpd compiled to mpm_prefork_module had the same issues as mpm_event_module.

Edit 2:
mod_ruid2 was the culprit (despite not giving any relevant errors) and was failing event with mod_php/mpm_prefork. Disabling mod_ruid2 in either mode resolves the issue.
 
Last edited:

Active8

Verified User
Joined
Jul 13, 2013
Messages
210
A wild guess, you have used a minimal centos 8 install without any LAMP stack ?
 

Willis

Verified User
Joined
Dec 31, 2005
Messages
30
Correct, I have used a CentOS 8 netinstall using server-only. Apache httpd was installed fully by DA in ./setup.sh auto
Apache httpd is running, but the worker processes are aborting.

Install process was:
Install CentOS 8 netinstall server only on KVM.
Install CentOS 8 pre-reqs for DA.
Install DA via ./setup.sh auto
After successful install, attempt to visit http://[ip] and see failures in error log.

Rebuilding httpd through custom build appears to show no issues, and continues to fail in the same way.
 

Willis

Verified User
Joined
Dec 31, 2005
Messages
30
I've just re-issued a custom build via ./build apache and the log is attached just for better reference. I see no relevant errors in the build log.
 

Attachments

Willis

Verified User
Joined
Dec 31, 2005
Messages
30
My first install was without auto, the last two have been with. All 3 have ended with the same issue.
Yes, I have read the install guide and meet all the requirements very easily.
 

Active8

Verified User
Joined
Jul 13, 2013
Messages
210
mod_ruid2 is disabled ? and try php 7.3 instead of 7.2 as test
 

Willis

Verified User
Joined
Dec 31, 2005
Messages
30
mod_ruid2 is enabled by default and shows no error in the apache httpd build process (you can find it building mod_ruid2 in the build log I attached earlier). On my first install, I did in fact try both 7.2 and 7.3, both give the same error. I'm simply trying to serve the static index.html from /var/www/html/index.html and that is failing.

I will try compiling with mod_ruid2 disabled.
 

Willis

Verified User
Joined
Dec 31, 2005
Messages
30
Bingo, I am able to get a static file with mod_ruid2 disabled. Now the question is why there is no error about mod_ruid2.
 

Active8

Verified User
Joined
Jul 13, 2013
Messages
210
mod_ruid2 works not good with php-fpm , and it has no use anymore.
Its advised not to use this combination
 

Willis

Verified User
Joined
Dec 31, 2005
Messages
30
I was getting the same error with mod_ruid2 enabled and mpm-prefork/mod_php. DA defaults to mod_ruid2 enabled and mpm-prefork/mod_php.
 

Active8

Verified User
Joined
Jul 13, 2013
Messages
210
Yes mod_php and mod_ruid2 works ok but if you switch to php-fpm you must disable mod_ruid2
 

Willis

Verified User
Joined
Dec 31, 2005
Messages
30
No, I was getting the same process aborts on mod_php with mod_ruid2.

Like I said, ./setup.sh auto results in http 2.4 with mod_ruid2 and php_mod/mpm_prefork, and it gives the same exact process abort error on CentOS 8 for me.

I am considering this matter here closed, and I am moving the mod_ruid2 discussion to the CentOS 8 forum.
 
Top