Apache failure - no error in logs

SupermanInNY

Verified User
Joined
Sep 28, 2004
Messages
419
Hi All,

I have a server that is showing no symptoms of failure, except that the sites stop respond.
invoking a 'service httpd restart' brings them back.

# php -v
PHP 5.6.31 (cli) (built: Aug 11 2017 15:41:09)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v10.0.0 (), Copyright (c) 2002-2017, by ionCube Ltd.
with Zend Guard Loader v3.3, Copyright (c) 1998-2014, by Zend Technologies
with Suhosin v0.9.38, Copyright (c) 2007-2015, by SektionEins GmbH


t # php -i |more
phpinfo()
PHP Version => 5.6.31

System => Linux server.interuse.com 2.6.32-696.6.3.el6.x86_64 #1 SMP Wed Jul 12
14:17:22 UTC 2017 x86_64
Build Date => Aug 11 2017 15:40:52
Configure Command => './configure' '--prefix=/usr/local/php56' '--program-suff
ix=56' '--with-config-file-scan-dir=/usr/local/php56/lib/php.conf.d' '--with-cur
l=/usr/local/lib' '--with-gd' '--enable-gd-native-ttf' '--with-gettext' '--with-
jpeg-dir=/usr/local/lib' '--with-freetype-dir=/usr/local/lib' '--with-libxml-dir
=/usr/local/lib' '--with-kerberos' '--with-openssl' '--with-mcrypt' '--with-mhas
h' '--with-mysql=mysqlnd' '--with-mysql-sock=/var/lib/mysql/mysql.sock' '--with-
mysqli=mysqlnd' '--with-pcre-regex=/usr/local' '--with-pdo-mysql=mysqlnd' '--wit
h-pear' '--with-png-dir=/usr/local/lib' '--with-xsl' '--with-zlib' '--with-zlib-
dir=/usr/local/lib' '--enable-zip' '--with-iconv=/usr/local' '--enable-bcmath' '
--enable-calendar' '--enable-ftp' '--enable-sockets' '--enable-soap' '--enable-m
bstring' '--with-icu-dir=/usr/local/icu' '--enable-intl' '--enable-exif'
Server API => Command Line Interface
Virtual Directory Support => disabled
Configuration File (php.ini) Path => /usr/local/php56/lib
Loaded Configuration File => /usr/local/php56/lib/php.ini
Scan this dir for additional .ini files => /usr/local/php56/lib/php.conf.d
Additional .ini files parsed => /usr/local/php56/lib/php.conf.d/10-directadmin.i



# httpd -v
Server version: Apache/2.4.27 (Unix)
Server built: Aug 7 2017 00:18:47

As you can see the error_log below doesn't show any issues till we invoke service restart.

[Wed Sep 06 03:30:16.687647 2017] [:error] [pid 20034:tid 140544810333952] [client 74.208.165.33:61961] [client 74.208.165.33] ModSecurity: Access denied with code 403 (phase 2). Match of "endsWith /modules/paypal/express_checkout/payment.php" against "REQUEST_FILENAME" required. [file "/usr/local/cwaf/rules/02_Global_Generic.conf"] [line "24"] [id "211120"] [rev "10"] [msg "COMODO WAF: Remote File Inclusion Attack||www.inaatrxgaomedix.coml|F|2"] [data "Matched Data: ftp://premielo:[email protected]/envi.php? found within REQUEST_FILENAME: /index2.php"] [severity "CRITICAL"] [tag "CWAF"] [tag "Generic"] [hostname "www.inaatrxgaomedix.com"] [uri "/index2.php"] [unique_id "Wa9BmD5ajVAAAE5CD7gAAAAZ"]
[Wed Sep 06 03:30:16.696625 2017] [:error] [pid 20034:tid 140544810333952] [client 74.208.165.33:61961] [client 74.208.165.33] ModSecurity: Audit log: Failed to create subdirectories: /var/log/modsec_audit/apache/20170906/20170906-0330 (Permission denied) [hostname "www.inaatrxgaomedix.com"] [uri "/index2.php"] [unique_id "Wa9BmD5ajVAAAE5CD7gAAAAZ"]
[Wed Sep 06 11:27:38.533884 2017] [core:warn] [pid 22957:tid 140545181153216] AH00045: child process 20852 still did not exit, sending a SIGTERM
[Wed Sep 06 11:27:38.533977 2017] [core:warn] [pid 22957:tid 140545181153216] AH00045: child process 21483 still did not exit, sending a SIGTERM
[Wed Sep 06 11:27:40.536299 2017] [mpm_event:notice] [pid 22957:tid 140545181153216] AH00491: caught SIGTERM, shutting down
[Wed Sep 06 11:27:47.183782 2017] [ssl:warn] [pid 25761:tid 140387976189888] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
[Wed Sep 06 11:27:47.186460 2017] [suexec:notice] [pid 25761:tid 140387976189888] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Sep 06 11:27:47.186501 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/) configured.
[Wed Sep 06 11:27:47.186512 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity: APR compiled version="1.6.2"; loaded version="1.6.2"
[Wed Sep 06 11:27:47.186518 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity: PCRE compiled version="8.20 "; loaded version="8.20 2011-10-21"
[Wed Sep 06 11:27:47.186523 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity: LIBXML compiled version="2.9.3"
[Wed Sep 06 11:27:47.186527 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity: Original server signature: Apache/2.4.27 (Unix) OpenSSL/1.0.1e-fips
[Wed Sep 06 11:27:47.186531 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Wed Sep 06 11:27:48.036001 2017] [ssl:warn] [pid 25765:tid 140387976189888] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
[Wed Sep 06 11:27:48.050160 2017] [lbmethod_heartbeat:notice] [pid 25765:tid 140387976189888] AH02282: No slotmem from mod_heartmonitor
[Wed Sep 06 11:27:48.063284 2017] [mpm_event:notice] [pid 25765:tid 140387976189888] AH00489: Apache/2.4.27 (Unix) OpenSSL/1.0.1e-fips Protected by COMODO WAF mod_fcgid/2.3.9 configured -- resuming normal operations
[Wed Sep 06 11:27:48.063349 2017] [core:notice] [pid 25765:tid 140387976189888] AH00094: Command line: '/usr/sbin/httpd'
(




Any pointers for where else to look?

Thanks,

-Sup.
 
What about this?
02_Global_Generic.conf"] [line "24"] [id "211120"] [rev "10"] [msg "COMODO WAF: Remote File Inclusion Attack||www.inaatrxgaomedix.coml|F|2"] [data "Matched Data: ftp://[email protected]/envi.php? found within REQUEST_FILENAME: /index2.php"] [severity "CRITICAL"] [tag "CWAF"] [tag "Generic"] [hostname "www.inaatrxgaomedix.com"] [uri "/index2.php"] [unique_id "Wa9BmD5ajVAAAE5CD7gAAAAZ"]
 
that's an indication of the blocked attack I think.
I don't see this a "crash" of the apache server.
Its not a SegFault error msg or something like that.
Also, notice the timestamp diff between the events.
 
Check your serverload. Warnings like:

[Wed Sep 06 11:27:38.533884 2017] [core:warn] [pid 22957:tid 140545181153216] AH00045: child process 20852 still did not exit, sending a SIGTERM

might indicate that php is not handling (some of?) the request fast enough. So requests get queued -> take longer -> and before you know it you have the maximum apache processes running ( as configured ). But if the processes are all in use and waiting for e.g. a php request to end, they cannot server any other request. That would explain why it looks like apache stopped working and it works again if you restart apache (as that kill's all processes).

thing to look into:

- server load
- apache memory usage
- scripts that run too long
- cronjobs that start php scripts while not having finished the previous run
- reverse shell scripts that keep connections open.
 
So where is my starting point?

1. I have no high server load
2. How to check for apache memory usage and utilization? run top?
3. no cronjobs in the background
4. how to identify reverse shell scripts?


Where should I make changes?

httpd-mpm.conf

Increase values of various defaults?
 
Hi,

I got this issue one time... The problem was one site is taking all available connections and make apache stuck (running, no error but not serving any request) for an half hour.

Here is how I have solved the problem:

Check how many connection by IP to port 80

netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head

If and IP have a lot of connexion to port 80, check in the apache logs to find witch site is viewed by this IP:

grep -rnw '/var/log/httpd' -e '000.000.000.000'

Suspend the abusive user/site .

Sorry for my bad english... i'm a native french speaker :)
 
Last edited:
Hi,

I got this issue one time... The problem was one site is taking all available connections and make apache stuck (running, no error but not serving any request) for an half hour.

Here is how I have solved the problem:

Check how many connection by IP to port 80



If and IP have a lot of connexion to port 80, check in the apache logs to find witch site is viewed by this IP:



Suspend the abusive user/site .

Sorry for my bad english... i'm a native french speaker :)

Your English is fine !
Thank you for trying to assist me with this.
Unfortunately, I'm already familiar with this solution, I have it as a file called dos-attack.sh and I run it when there is an attack,. so I can block with CSF and of course identify who is being targeted.
But nothing is pulling up as a 'big' number of connections :(
 
If you get lots of DDOS, you need a firewall (ex. Iptables & fail2ban) to block them and get rid of high server load.
 
If you get lots of DDOS, you need a firewall (ex. Iptables & fail2ban) to block them and get rid of high server load.

Thanks, we already have CSF enabled by default and actually we have very low server loads.
Nothing is obvious in this scenario.
 
Back
Top