APACHE 2.4 + php-fpm not closing old thread and 100% cPU usage

M

Mangas23

Verified User
Joined
Aug 31, 2006
Messages
104
Hello, i need help please, i find no solution to resolve this problem:
Recently i change server for new one in centos 6 x64 and reinstall all with custombuild 2. All look good, but recently i see problem to received email, and when i check my server i see my xeon 8 core at 100% !!!! all core, so in averload, and email not send to receiver.
So i check my process, i fund no attack, only old process of httpd not closed.
I try modify httpd-mpm.conf (event section) with no result.

Someone already had this problem ?? and how to resolve this ??

PS: i dont want nginx

I send you screenshot when problem come back ( i just restart httpd server)

Thanks
 
screenshot:
 

Attachments

  • 1.jpg
    1.jpg
    245.7 KB · Views: 354
  • 2.jpg
    2.jpg
    378.5 KB · Views: 323
problem not come php-fpm, because i try all (suphp fastcgi) and same problem.
 
It looks to me that maybe you're under flood attack since you do have free memory and just cpu overload...

Does you have a firewall?

Regards
 
Yes shure ;) with fail2ban to check log and ban ip, but i see nothing in log :( Or i dont search well ;)
with this rules for GET/POST
[apache-get-post]

enabled = true
filter = apache-get-post
action = iptables-multiport[name=Apache-get-post, port="http,https"]
sendmail-whois[name=Apache-get-post, [email protected], [email protected]]
logpath = /var/log/httpd/access_log
/var/log/httpd/domains/*.com.log
/var/log/httpd/domains/*.org.log
/var/log/httpd/domains/*.net.log
/var/log/httpd/domains/*.fr.log
/var/log/httpd/domains/*.us.log
/var/log/httpd/domains/*.biz.log
/var/log/httpd/domains/*.ca.log
/var/log/httpd/domains/*.info.log
/var/log/httpd/domains/*.eu.log
findtime = 10
maxretry = 40

Me too i thinking at start that flood attack, but why thread i launch yesterday can see it this morning in /server-status ??
When i close my web client apache must close my thread after timeout, no ??
And when i check log of domain i see only regular request of POST/GET and this in error :
ModSecurity: Access denied with code 403 (phase 1). String match "HTTP/1.0" at REQUEST_PROTOCOL. [file "/etc/modsecurity.d/cwaf_01.conf"] [line "232"] [id "210300"] [msg "COMODO WAF: Expect Header Not Allowed for HTTP 1.0."] [data "HTTP/1.0"] [severity "NOTICE"] [hostname "stbarthyachtclub.com"] [uri "/index.php"] [unique_id "VMzE16dyVwAAAD8snlYAAAPP"]

but not with excessive try
 
So all working well before i put apache 2.4, on my old server with centos 5 32bits and custombuild 1.2, apache 2.2 no problème i configure php same on new and old. personaly i dont understand :p
 
I definitly doubt it depend on software version and releases... And i'm not sure fail2ban is what you need, have you tryed use CSF Firewall?

Also, keep in mind that may be a script on local server (maybe a faulty account) which is using php from command line or other code (bash) so you will not have any apache hint with those...

You may also try to use rkhunter

Regards
 
Did you check Apache server-status page? What status do those threads have? W? Check your MySQL/PHP stack, probably you need to review your MySQL tables and queries.
 
not script in user homedir, i already check it
I had already check /server-status

ok i think i have fund problem:
0-0 6119 0/1/1 _ 0.08 715 1 0.0 0.00 0.00 64.20.55.236 shared.domain:80 GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1
0-0 6119 0/1/1 _ 0.08 715 3 0.0 0.01 0.01 64.20.55.236 shared.domain:80 GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1
0-0 6119 0/1/1 _ 0.08 714 2 0.0 0.01 0.01 64.20.55.236 shared.domain:80 GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1
0-0 6119 0/1/1 _ 0.08 714 0 0.0 0.04 0.04 64.20.55.236 shared.domain:80 GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1

but some right request do same.
With what i fund in log i think that all time there are lot of request GET or POST (like for CMS) httpd take max cpu usage, problem with cache ? with HD ? I dont understand :p

9-0 23815 0/2/2 _ 0.22 9 0 0.0 0.99 0.99 78.222.221.206 www.confortmedical.net:80 GET /images/fauteuil%20roulant_small.JPG HTTP/1.1
9-0 23815 0/2/2 _ 0.22 9 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /images/saint%20cloud/images/electrode/electrodea.gif HTTP/
9-0 23815 0/2/2 _ 0.22 9 0 0.0 0.01 0.01 78.222.221.206 www.confortmedical.net:80 GET /_derived/publication.htm_cmp_copie-de-axe3010_vbtn.gif HTT
9-0 23815 0/2/2 _ 0.22 9 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/service%20ags.htm_cmp_copie-de-axe3010_vbtn.gif H
9-0 23815 0/2/2 _ 0.22 9 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /images/douche%20au%20lit/douche%20au%20lit%203.jpg HTTP/1.
9-0 23815 0/2/2 _ 0.22 9 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /images/tiralo%203.JPG HTTP/1.1
9-0 23815 0/2/2 _ 0.22 8 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/professionnelle.htm_cmp_copie-de-axe3010_vbtn_a.g
9-0 23815 0/2/2 _ 0.22 8 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/search.htm_cmp_copie-de-axe3010_vbtn_a.gif HTTP/1
10-0 23863 0/2/2 _ 377.07 20 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/search.htm_cmp_copie-de-axe3010_vbtn_a.gif HTTP/1
10-0 23863 0/2/2 _ 377.07 20 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/relaxation%20et%20massage.htm_cmp_copie-de-axe301
10-0 23863 0/2/2 _ 377.18 19 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/kinesitherapeutes.htm_cmp_copie-de-axe3010_vbtn_a
10-0 23863 0/3/3 _ 377.18 19 0 0.0 0.01 0.01 78.222.221.206 www.confortmedical.net:80 GET /_derived/publication.htm_cmp_copie-de-axe3010_vbtn_a.gif H
10-0 23863 0/3/3 _ 377.21 19 0 0.0 0.01 0.01 78.222.221.206 www.confortmedical.net:80 GET /_derived/ext1032_cmp_copie-de-axe3010_vbtn_a.gif HTTP/1.1
10-0 23863 0/3/3 _ 377.39 19 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /favicon.ico HTTP/1.1
10-0 23863 0/2/2 _ 377.50 19 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /favicon.ico HTTP/1.1
10-0 23863 0/3/3 _ 387.86 9 0 0.0 0.01 0.01 78.222.221.206 www.confortmedical.net:80 GET /images/lit_small.JPG HTTP/1.1
10-0 23863 0/3/3 _ 387.97 9 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/site_guadeloupeen_pour_une_vie_a.htm_cmp_copie-de
10-0 23863 0/0/0 L 0.00 396 0 0.0 0.00 0.00 78.222.221.206 localhost:80 GET /server-status HTTP/1.1
10-0 23863 0/3/3 _ 388.19 8 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/relaxation%20et%20massage.htm_cmp_copie-de-axe301
10-0 23863 0/3/3 _ 388.08 9 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /images/Tiralo.JPG HTTP/1.1
10-0 23863 0/3/3 _ 388.30 8 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/location.htm_cmp_copie-de-axe3010_vbtn_a.gif HTTP
10-0 23863 0/3/3 _ 388.42 8 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/medecin.htm_cmp_copie-de-axe3010_vbtn_a.gif HTTP/
10-0 23863 0/3/3 _ 388.52 8 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/home_cmp_copie-de-axe3010_vbtn_a.gif HTTP/1.1
10-0 23863 0/3/3 _ 388.63 8 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/incontinence.htm_cmp_copie-de-axe3010_vbtn.gif HT
10-0 23863 0/3/3 _ 388.74 8 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/infirmieres.htm_cmp_copie-de-axe3010_vbtn.gif HTT
10-0 23863 0/3/3 _ 388.86 8 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /promotion.gif HTTP/1.1
10-0 23863 0/2/2 _ 376.84 20 13 0.0 0.01 0.01 78.222.221.206
10-0 23863 0/2/2 _ 376.84 20 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /images/tapis%20d%27entrainement.JPG HTTP/1.1
10-0 23863 0/2/2 _ 376.87 20 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/professionnelle.htm_cmp_copie-de-axe3010_vbtn_a.g
10-0 23863 0/2/2 _ 376.94 20 0 0.0 0.01 0.01 78.222.221.206 www.confortmedical.net:80 GET /_derived/Materiel%20a%20domicil.htm_cmp_copie-de-axe3010_v
10-0 23863 0/2/2 _ 376.95 20 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/aide_a_la_toilette.htm_cmp_copie-de-axe3010_vbtn_
10-0 23863 0/2/2 _ 376.98 20 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/fitness.htm_cmp_copie-de-axe3010_vbtn_a.gif HTTP/
10-0 23863 0/3/3 _ 377.10 20 0 0.0 0.00 0.00 78.222.221.206 www.confortmedical.net:80 GET /_derived/infirmieres.htm_cmp_copie-de-axe3010_vbtn_a.gif H

all this are my ip and i only go on site with lot of missing image in index page. This site not infected i check it. (i disable site, user dont use it but it's same for all CMS :( )
Someone have an idea to resolve this ?
Or tell me how i can downgrade custombuild 2.0 to 1.2 (if work with centos 6 64bits) and resinstall all soft.
Thanks
 
Last edited:
no custom page used, and i prefer go back to apache 2.2 that work perfectly since 2005 without nginx, i prefer downgrade to same config on old server ;)
 
All known CMS'es uses mod_rewrite rules to trigger a php script on 404 error. Thus on every missing image a PHP script is executed with probably tens/hundreds requests to MySQL if no cache is used.
 
the problem it's not 404, because when you go on site with no 404 but lot of GET that do same. But if you tell me with nginx i can resolve this i test ;)
But why i had no problem before with apache 2.2 suphp mysql etc... and same user, site etc... ?? I just change apache, zend by opcache and nothing else i think, I just want understand :)
 
Last edited:
Apache 2.2 as far as i remember i slower than Apache 2.4

I do use 2.4 since more than a year if i remember correctly, and definitly i had a powerup on speed.. not a powerdown.. the problem is somewhere else.. definitly not on software versions

Regards
 
Mangas23 said:
But if you tell me with nginx i can resolve this i test
Click to expand...

I won't guarantee you that unless I see the server from inside and do configuration by my hands.

Mangas23 said:
But why i had no problem before with apache 2.2 suphp mysql etc... and same user, site etc... ??
Click to expand...

Who knows? Not me. There might be many other aspects which stay beyond your observation probably. Do you measure your bandwidth, traffic volume? Daily visits? Queries to MySQL? I guess you don't; if you did that you could probably see on graphs what changed.


Broken MySQL tables might bring issues. Did you try to check and fix it?


Is that a VPS? Do you monitor IO/wait and latency?


Is that hardware server? Do you monitor health of your HDDs and IO/wait and latency?


Do you or your customers recently begin to buy ADs in Internet? Did they start any promotions?


I agree with Andrea versions of software hardly are the answer.
 
yes i shure apache 2.4 are better ;) so perhaps php-fpm, before i use suphp with only one php version, no i use php-fpm with php 5.3 and 5.5 installed.
I check traffic, visit etc and nothing wrong, no more than every day.
for HDD i check soon.
 
Last edited:
I miss to say i put this in configure.php53
#!/bin/sh
./configure \
--prefix=/usr/local/php53 \
--program-suffix=53 \
--enable-fpm \
--with-config-file-scan-dir=/usr/local/php53/lib/php.conf.d \
--with-curl=/usr/local/lib \
--with-gd \
--enable-gd-native-ttf \
--with-ttf \
--with-gettext \
--with-jpeg-dir=/usr/local/lib \
--with-freetype-dir=/usr/local/lib \
--with-libxml-dir=/usr/local/lib \
--with-kerberos \
--with-openssl \
--with-mcrypt \
--with-mhash \
--with-mysql=mysqlnd \
--with-mysql-sock=/var/lib/mysql/mysql.sock \
--with-mysqli=mysqlnd \
--with-pcre-regex=/usr/local \
--with-pdo \
--enable-pdo \
--with-pdo-mysql=mysqlnd \
--with-sqlite \
--with-pdo-sqlite \
--with-pear \
--with-png-dir=/usr/local/lib \
--with-xsl \
--with-zlib \
--with-zlib-dir=/usr/local/lib \
--enable-zip \
--with-iconv=/usr/local \
--enable-bcmath \
--enable-calendar \
--enable-ftp \
--enable-sockets \
--enable-soap \
--enable-mbstring \
--enable-magic-quotes \
--enable-track-vars \
--disable-posix \
--with-icu-dir=/usr/local/icu \
--enable-intl

I add pdo it's perhaps for this ?
 
Ok i have find the problem, remove modsecurity and MY GOD that work !!!!!!! :p without CPU flood :)
So now why ? :) Perhaps that will be a good idea to find what, because i prefer use this mod if possible :)
 
Last edited:
Try changing "SecRequestBodyLimit 13107200" to "SecRequestBodyLimit 131072" in /etc/httpd/conf/extra/httpd-modsecurity.conf and see if that helps. Replace "SecRequestBodyLimitAction Reject" with "SecRequestBodyLimitAction ProcessPartial".
 
Last edited:
You must log in or register to reply here.
Back
Top