Results 1 to 11 of 11

Thread: Apache failure - no error in logs

  1. #1
    Join Date
    Sep 2004
    Posts
    344

    Apache failure - no error in logs

    Hi All,

    I have a server that is showing no symptoms of failure, except that the sites stop respond.
    invoking a 'service httpd restart' brings them back.

    # php -v
    PHP 5.6.31 (cli) (built: Aug 11 2017 15:41:09)
    Copyright (c) 1997-2016 The PHP Group
    Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v10.0.0 (), Copyright (c) 2002-2017, by ionCube Ltd.
    with Zend Guard Loader v3.3, Copyright (c) 1998-2014, by Zend Technologies
    with Suhosin v0.9.38, Copyright (c) 2007-2015, by SektionEins GmbH


    t # php -i |more
    phpinfo()
    PHP Version => 5.6.31

    System => Linux server.interuse.com 2.6.32-696.6.3.el6.x86_64 #1 SMP Wed Jul 12
    14:17:22 UTC 2017 x86_64
    Build Date => Aug 11 2017 15:40:52
    Configure Command => './configure' '--prefix=/usr/local/php56' '--program-suff
    ix=56' '--with-config-file-scan-dir=/usr/local/php56/lib/php.conf.d' '--with-cur
    l=/usr/local/lib' '--with-gd' '--enable-gd-native-ttf' '--with-gettext' '--with-
    jpeg-dir=/usr/local/lib' '--with-freetype-dir=/usr/local/lib' '--with-libxml-dir
    =/usr/local/lib' '--with-kerberos' '--with-openssl' '--with-mcrypt' '--with-mhas
    h' '--with-mysql=mysqlnd' '--with-mysql-sock=/var/lib/mysql/mysql.sock' '--with-
    mysqli=mysqlnd' '--with-pcre-regex=/usr/local' '--with-pdo-mysql=mysqlnd' '--wit
    h-pear' '--with-png-dir=/usr/local/lib' '--with-xsl' '--with-zlib' '--with-zlib-
    dir=/usr/local/lib' '--enable-zip' '--with-iconv=/usr/local' '--enable-bcmath' '
    --enable-calendar' '--enable-ftp' '--enable-sockets' '--enable-soap' '--enable-m
    bstring' '--with-icu-dir=/usr/local/icu' '--enable-intl' '--enable-exif'
    Server API => Command Line Interface
    Virtual Directory Support => disabled
    Configuration File (php.ini) Path => /usr/local/php56/lib
    Loaded Configuration File => /usr/local/php56/lib/php.ini
    Scan this dir for additional .ini files => /usr/local/php56/lib/php.conf.d
    Additional .ini files parsed => /usr/local/php56/lib/php.conf.d/10-directadmin.i



    # httpd -v
    Server version: Apache/2.4.27 (Unix)
    Server built: Aug 7 2017 00:18:47

    As you can see the error_log below doesn't show any issues till we invoke service restart.

    [Wed Sep 06 03:30:16.687647 2017] [:error] [pid 20034:tid 140544810333952] [client 74.208.165.33:61961] [client 74.208.165.33] ModSecurity: Access denied with code 403 (phase 2). Match of "endsWith /modules/paypal/express_checkout/payment.php" against "REQUEST_FILENAME" required. [file "/usr/local/cwaf/rules/02_Global_Generic.conf"] [line "24"] [id "211120"] [rev "10"] [msg "COMODO WAF: Remote File Inclusion Attack||www.inaatrxgaomedix.coml|F|2"] [data "Matched Data: ftp://premieloremielo@www.premieloftet.no/envi.php? found within REQUEST_FILENAME: /index2.php"] [severity "CRITICAL"] [tag "CWAF"] [tag "Generic"] [hostname "www.inaatrxgaomedix.com"] [uri "/index2.php"] [unique_id "Wa9BmD5ajVAAAE5CD7gAAAAZ"]
    [Wed Sep 06 03:30:16.696625 2017] [:error] [pid 20034:tid 140544810333952] [client 74.208.165.33:61961] [client 74.208.165.33] ModSecurity: Audit log: Failed to create subdirectories: /var/log/modsec_audit/apache/20170906/20170906-0330 (Permission denied) [hostname "www.inaatrxgaomedix.com"] [uri "/index2.php"] [unique_id "Wa9BmD5ajVAAAE5CD7gAAAAZ"]
    [Wed Sep 06 11:27:38.533884 2017] [core:warn] [pid 22957:tid 140545181153216] AH00045: child process 20852 still did not exit, sending a SIGTERM
    [Wed Sep 06 11:27:38.533977 2017] [core:warn] [pid 22957:tid 140545181153216] AH00045: child process 21483 still did not exit, sending a SIGTERM
    [Wed Sep 06 11:27:40.536299 2017] [mpm_event:notice] [pid 22957:tid 140545181153216] AH00491: caught SIGTERM, shutting down
    [Wed Sep 06 11:27:47.183782 2017] [ssl:warn] [pid 25761:tid 140387976189888] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
    [Wed Sep 06 11:27:47.186460 2017] [suexec:notice] [pid 25761:tid 140387976189888] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
    [Wed Sep 06 11:27:47.186501 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/) configured.
    [Wed Sep 06 11:27:47.186512 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity: APR compiled version="1.6.2"; loaded version="1.6.2"
    [Wed Sep 06 11:27:47.186518 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity: PCRE compiled version="8.20 "; loaded version="8.20 2011-10-21"
    [Wed Sep 06 11:27:47.186523 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity: LIBXML compiled version="2.9.3"
    [Wed Sep 06 11:27:47.186527 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity: Original server signature: Apache/2.4.27 (Unix) OpenSSL/1.0.1e-fips
    [Wed Sep 06 11:27:47.186531 2017] [:notice] [pid 25761:tid 140387976189888] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
    [Wed Sep 06 11:27:48.036001 2017] [ssl:warn] [pid 25765:tid 140387976189888] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
    [Wed Sep 06 11:27:48.050160 2017] [lbmethod_heartbeat:notice] [pid 25765:tid 140387976189888] AH02282: No slotmem from mod_heartmonitor
    [Wed Sep 06 11:27:48.063284 2017] [mpm_event:notice] [pid 25765:tid 140387976189888] AH00489: Apache/2.4.27 (Unix) OpenSSL/1.0.1e-fips Protected by COMODO WAF mod_fcgid/2.3.9 configured -- resuming normal operations
    [Wed Sep 06 11:27:48.063349 2017] [core:notice] [pid 25765:tid 140387976189888] AH00094: Command line: '/usr/sbin/httpd'
    (




    Any pointers for where else to look?

    Thanks,

    -Sup.

  2. #2
    Join Date
    May 2014
    Location
    Netherlands Germany
    Posts
    278
    What about this?
    02_Global_Generic.conf"] [line "24"] [id "211120"] [rev "10"] [msg "COMODO WAF: Remote File Inclusion Attack||www.inaatrxgaomedix.coml|F|2"] [data "Matched Data: ftp://premieloremielo@www.premieloftet.no/envi.php? found within REQUEST_FILENAME: /index2.php"] [severity "CRITICAL"] [tag "CWAF"] [tag "Generic"] [hostname "www.inaatrxgaomedix.com"] [uri "/index2.php"] [unique_id "Wa9BmD5ajVAAAE5CD7gAAAAZ"]
    DUTCH GERMAN, GERMAN DUTCH

  3. #3
    Join Date
    Sep 2004
    Posts
    344
    that's an indication of the blocked attack I think.
    I don't see this a "crash" of the apache server.
    Its not a SegFault error msg or something like that.
    Also, notice the timestamp diff between the events.

  4. #4
    Join Date
    Jul 2007
    Posts
    156
    Check your serverload. Warnings like:

    [Wed Sep 06 11:27:38.533884 2017] [core:warn] [pid 22957:tid 140545181153216] AH00045: child process 20852 still did not exit, sending a SIGTERM

    might indicate that php is not handling (some of?) the request fast enough. So requests get queued -> take longer -> and before you know it you have the maximum apache processes running ( as configured ). But if the processes are all in use and waiting for e.g. a php request to end, they cannot server any other request. That would explain why it looks like apache stopped working and it works again if you restart apache (as that kill's all processes).

    thing to look into:

    - server load
    - apache memory usage
    - scripts that run too long
    - cronjobs that start php scripts while not having finished the previous run
    - reverse shell scripts that keep connections open.

  5. #5
    Join Date
    Sep 2004
    Posts
    344
    So where is my starting point?

    1. I have no high server load
    2. How to check for apache memory usage and utilization? run top?
    3. no cronjobs in the background
    4. how to identify reverse shell scripts?


    Where should I make changes?

    httpd-mpm.conf

    Increase values of various defaults?

  6. #6
    Join Date
    Feb 2008
    Location
    Québec, Canada
    Posts
    148
    Hi,

    I got this issue one time... The problem was one site is taking all available connections and make apache stuck (running, no error but not serving any request) for an half hour.

    Here is how I have solved the problem:

    Check how many connection by IP to port 80

    netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head
    If and IP have a lot of connexion to port 80, check in the apache logs to find witch site is viewed by this IP:

    grep -rnw '/var/log/httpd' -e '000.000.000.000'
    Suspend the abusive user/site .

    Sorry for my bad english... i'm a native french speaker
    Last edited by webaltern; 09-08-2017 at 03:12 PM.
    Philippe Robert
    --

  7. #7
    Join Date
    Jul 2007
    Posts
    156
    Maybe this will help:

    https://github.com/jgmdev/ddos-deflate

    If not, hire a serverguy to check things. There are a few on this forum.

  8. #8
    Join Date
    Sep 2004
    Posts
    344
    Quote Originally Posted by webaltern View Post
    Hi,

    I got this issue one time... The problem was one site is taking all available connections and make apache stuck (running, no error but not serving any request) for an half hour.

    Here is how I have solved the problem:

    Check how many connection by IP to port 80



    If and IP have a lot of connexion to port 80, check in the apache logs to find witch site is viewed by this IP:



    Suspend the abusive user/site .

    Sorry for my bad english... i'm a native french speaker
    Your English is fine !
    Thank you for trying to assist me with this.
    Unfortunately, I'm already familiar with this solution, I have it as a file called dos-attack.sh and I run it when there is an attack,. so I can block with CSF and of course identify who is being targeted.
    But nothing is pulling up as a 'big' number of connections

  9. #9
    Join Date
    Sep 2004
    Posts
    344
    Quote Originally Posted by sysdev View Post
    Maybe this will help:

    https://github.com/jgmdev/ddos-deflate

    If not, hire a serverguy to check things. There are a few on this forum.
    Thanks for the ddos-deflate utility.
    I installed it now... and hopefully it will track and capture such activity.
    Within 3 days I'll see either it is the source of the problem (a DDoS attack) or something else.

    Thanks,

    -Sup.

  10. #10
    Join Date
    Sep 2016
    Posts
    22
    If you get lots of DDOS, you need a firewall (ex. Iptables & fail2ban) to block them and get rid of high server load.

  11. #11
    Join Date
    Sep 2004
    Posts
    344
    Quote Originally Posted by jwillberg View Post
    If you get lots of DDOS, you need a firewall (ex. Iptables & fail2ban) to block them and get rid of high server load.
    Thanks, we already have CSF enabled by default and actually we have very low server loads.
    Nothing is obvious in this scenario.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •