apache 2.4.49,2.4.52 strange unreachable after pass few hours

[Ohm J]

Centos 8 with Nginx_Apache
apache with mod_maxminddb
(mod_maxminddb not recomplier)

I just post here to anyone got same problem with me,

in now I temporary back to 2.4.48
problem start after update to 2.4.49

first I try to find log
/var/log/httpd/*

test with "pstack {PID}"
not show any strange thing

try to curl to "127.0.0.1:8080"
unreachable to ......

non any relate log here


and "service httpd status -l"
it's show running fine,

no any sync attack, and cpu, ram be fine too,
Total requests still show only 1, so not relate to too much resouce usage

but only strang thing I can see,


when on work normally

CGroup: /system.slice/httpd.service
├─11111 /usr/sbin/httpd -DFOREGROUND
└─22222 /usr/sbin/httpd -DFOREGROUND

when got problem

CGroup: /system.slice/httpd.service
├─11111 /usr/sbin/httpd -DFOREGROUND

so apache can't generate worker

###UPDATE 20-09-2021
From my some test

when starting and running normally

service httpd status -l

"Total requests" count normal, Example: count to 600K request

pass few hours it count down to "1" and other analytic ( eg. req/s ) down to "0"
look like it crash without report any log

last restart time still show on first started date
this is not restart, but crash itselt


###2021-12-26

add 2.4.52 to this issued

 

[kebirhost]

Hello,

I have this issue on my 50 servers too. I am using centos 7 and apache_nginx. Apache seems to work but Web sites are not working and we cannot see any connection from:

elinks http://localhost/server-status


There should a problem.

Thanks,
Melih

 

[wattie]

Same here on FreeBSD. Here is what happens in the logs:

var/log/messages: Sep 18 09:14:11 srv2 kernel: sonewconn: pcb 0xfffff80558b603a0: Listen queue overflow: 768 already in queue awaiting acceptance (750 oc currences)

httpd error log: [Sat Sep 18 09:14:56.389543 2021] [mpm_event:notice] [pid 85707:tid 34408062976] AH00491: caught SIGTERM, shutting down

HTTPD was running but not generating workers as you said. I read around the forums and fount a solution to increase the ServerLimit and MaxRequestWorkers. I did yesterday and so far it did not fall down, but I am still monitoring.

 

[Ohm J]

increase the ServerLimit and MaxRequestWorkers.

I already do this settings and work fine on 2.4.48

 

[anas_xrt]

I already do this settings and work fine on 2.4.48

How do you determine which number should be right for ServerLimit and MaxRequestWorkers that happen on 2.4.49 when I just move back to 2.4.48 with using default from Directadmin.

<IfModule mpm_event_module>
    StartServers             6
    MinSpareThreads         32
    MaxSpareThreads        128
    ThreadsPerChild         64
    ServerLimit             32
    MaxRequestWorkers     2048
    MaxConnectionsPerChild   10000
</IfModule>

*actually I found this article https://medium.com/@sbuckpesch/apac...-optimization-step-by-step-guide-1bfecf161534 but, since on the shared host is variable of number of users (180-200 users), so with default was great for me.

 

[Ohm J]

@anas_xrt
it depend on which you use

but don't try to increase unless you got warning from apache "error_log"

mpm_event
MaxRequestWorkers= ServerLimit * ThreadsPerChild

and should comparative with your hardware

in now I move back to 2.4.48

 

[anas_xrt]

@jamgames2
Meanwhile I think should be migrate back to 2.4.48 (as you did now) and observe for next release.

 

[nmb]

Coming from https://forum.directadmin.com/threads/httpd-2-4-49.64548/

Got same problem with one of my server with CentOS 7. nignx_apache setting. When problem appear, restart httpd fix it. No log to let me know what cause the problem though.

 

[Richard G]

Is everyone with this issue running nginx? Because it looks like this might be a apache/nginx combination issue.

We running 2 servers Centos 7 and one server Centos 8 with DA and we're not experiencing this issue on any of our servers until now. Updated couple of days ago, the day DA released it.

 

[ju5t]

We're not running Apache + Nginx and have the same issue.

 

[DanielP]

Same here, apache works 4-5 processes usually but when hangs there are only two just hangs and is green in monitoring

I downgraded too

 

[Active8]

Is everyone with this issue running nginx? Because it looks like this might be a apache/nginx combination issue.

We running 2 servers Centos 7 and one server Centos 8 with DA and we're not experiencing this issue on any of our servers until now. Updated couple of days ago, the day DA released it.

Running Centos 6 (ELS) + Centos 7 + Almalinux 8 servers, all with Nginx + apache combination and we have no problems at all.
Maybe just lucky ? (knock on the wood :0 )

 

[anas_xrt]

Running Centos 6 (ELS) + Centos 7 + Almalinux 8 servers, all with Nginx + apache combination and we have no problems at all.
Maybe just lucky ? (knock on the wood :0 )

any custom template or custom build for those Nginx_apache?

and today, I saw Directadmin team provide downgrade on custombuild 2.0 for this Apache as well.

 

[backusnetcom]

I have the same issues op CentOS 7/8, without nginx (but behind haproxy)

 

[Richard G]

Doesn't nGinx also work with proxy? Maybe it's cause by some proxy issue then?

 

[ju5t]

To be sure, we're not it using behind a proxy. We're using Apache in a standard setup.

 

[Richard G]

Which distro?

 

[ju5t]

All our servers are CentOS 8. We don't use custom configs for Apache. We are using ModSecurity though, but have not seen anything pointing into that direction yet. As others have noted, logs have no entries explaining this particular issue.

We have downgraded some servers within 1 day. Everything else was downgraded yesterday. No issues since.

 

[Active8]

any custom template or custom build for those Nginx_apache?

Nothing custom, everything is build with the default values.

 

[anas_xrt]

Nothing custom, everything is build with the default values.

Hi, I have try Apache/2.4.49 and I found if I disable CSF (csf -x) the server will survive. Do you have CSF firewall running on your server?