httpd stop suddenly

pppplus

pppplus

Verified User
Joined
Dec 19, 2008
Messages
523
Hi

On one of my servers, httpd stop working.
All services are OK, but websites do not work.

I can restart httpd and all is ok again.

In logs, I find nothing explaining this.
Have you idea of what I can search for ?

=> no overload, not too much processes


This happends 2-3 times a month.
 
Can you please provide us with the logfile around the time Apache stopped working?
 
Hi

Logs of /var/log/httpd/error_log
Code: 
[Fri Jul 19 16:32:57 2013] [warn] Rejected, too many connections in READ state from 189.237.143.75
[Fri Jul 19 16:32:58 2013] [warn] Rejected, too many connections in READ state from 189.237.143.75
[Fri Jul 19 16:32:58 2013] [warn] Rejected, too many connections in READ state from 189.237.143.75
[Fri Jul 19 16:33:00 2013] [warn] Rejected, too many connections in READ state from 189.237.143.75
[Fri Jul 19 16:33:02 2013] [warn] Rejected, too many connections in READ state from 59.95.34.220
[Fri Jul 19 16:33:02 2013] [warn] Rejected, too many connections in READ state from 59.95.34.220
[Fri Jul 19 16:33:02 2013] [warn] Rejected, too many connections in READ state from 59.95.34.220
[Fri Jul 19 16:33:03 2013] [warn] Rejected, too many connections in READ state from 189.237.143.75
[Fri Jul 19 16:37:26 2013] [warn] Rejected, too many connections in READ state from 114.46.215.90
[Fri Jul 19 16:37:27 2013] [warn] Rejected, too many connections in READ state from 114.46.215.90
[Fri Jul 19 16:40:31 2013] [warn] Rejected, too many connections in READ state from 113.254.62.17
[Fri Jul 19 16:45:43 2013] [warn] Rejected, too many connections in READ state from 122.26.78.164
[Fri Jul 19 16:48:04 2013] [warn] Rejected, too many connections in READ state from 94.38.50.111
[Fri Jul 19 16:48:04 2013] [warn] Rejected, too many connections in READ state from 94.38.50.111
[Fri Jul 19 16:48:04 2013] [warn] Rejected, too many connections in READ state from 94.38.50.111
[Fri Jul 19 16:48:34 2013] [warn] Rejected, too many connections in READ state from 85.60.145.102
[Fri Jul 19 16:48:34 2013] [warn] Rejected, too many connections in READ state from 85.60.145.102
[Fri Jul 19 16:48:34 2013] [warn] Rejected, too many connections in READ state from 85.60.145.102
[Fri Jul 19 16:49:30 2013] [error] [client 87.13.43.252] request failed: error reading the headers
[Fri Jul 19 16:50:47 2013] [warn] Rejected, too many connections in READ state from 190.204.81.191
[Fri Jul 19 16:50:47 2013] [warn] Rejected, too many connections in READ state from 190.204.81.191
[Fri Jul 19 16:50:48 2013] [warn] Rejected, too many connections in READ state from 190.204.81.191
[Fri Jul 19 16:52:08 2013] [warn] Rejected, too many connections in READ state from 113.254.62.17
[Fri Jul 19 17:09:10 2013] [notice] caught SIGTERM, shutting down

httpd stops at about 16h52
at 17h09, I restart httpd

logs of /var/log/message
Code: 
Jul 19 16:40:44 serveur8 named[8515]: client 74.125.189.23#34437: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:40:46 serveur8 named[8515]: client 74.125.189.20#58670: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:40:47 serveur8 named[8515]: client 74.125.189.19#54852: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:40:47 serveur8 named[8515]: client 74.125.189.20#48913: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:40:47 serveur8 named[8515]: client 74.125.189.16#62733: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:45:07 serveur8 named[8515]: client 88.190.253.33#45025: query (cache) 'europa-ptp.com/AAAA/IN' denied
Jul 19 16:45:07 serveur8 named[8515]: client 88.190.253.33#23143: query (cache) 'europa-ptp.com/AAAA/IN' denied
Jul 19 16:45:07 serveur8 named[8515]: client 88.190.253.33#5280: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 16:45:07 serveur8 named[8515]: client 88.190.253.33#39091: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 16:46:56 serveur8 named[8515]: client 213.136.96.2#14854: query (cache) 'www.europa-ptp.com/A/IN' denied
Jul 19 16:46:57 serveur8 named[8515]: client 213.136.96.37#36501: query (cache) 'www.europa-ptp.com/A/IN' denied
Jul 19 16:46:57 serveur8 named[8515]: client 213.136.96.2#37822: query (cache) 'www.europa-ptp.com/A/IN' denied
Jul 19 16:50:12 serveur8 named[8515]: client 74.125.189.23#56408: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:50:12 serveur8 named[8515]: client 74.125.189.22#56612: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:50:13 serveur8 named[8515]: client 74.125.189.16#62881: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:50:13 serveur8 named[8515]: client 74.125.189.18#60817: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:50:13 serveur8 named[8515]: client 74.125.189.21#52555: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:53:34 serveur8 named[8515]: client 157.56.96.9#40966: query (cache) 'webgamm.info/A/IN' denied
Jul 19 16:58:29 serveur8 named[8515]: client 74.125.189.21#58485: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:58:33 serveur8 named[8515]: client 74.125.189.23#52814: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:58:33 serveur8 named[8515]: client 74.125.189.16#36563: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:58:33 serveur8 named[8515]: client 74.125.189.17#37239: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:58:33 serveur8 named[8515]: client 74.125.189.18#44382: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 16:59:50 serveur8 named[8515]: client 65.55.37.38#23521: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 16:59:52 serveur8 named[8515]: client 66.249.75.99#43899: query (cache) 'www.europa-ptp.com/A/IN' denied
Jul 19 17:00:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:00:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:00:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:00:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:00:43 serveur8 named[8515]: client 157.56.96.9#33583: query (cache) 'www.bijouxetplus.com/A/IN' denied
Jul 19 17:01:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:01:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:01:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:01:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:01:18 serveur8 named[8515]: client 66.249.75.173#39859: query (cache) 'homactivity.eu/A/IN' denied
Jul 19 17:01:23 serveur8 lfd[14212]: SYSLOG check [YWmZu2P7ZCTJO3dG4HY]
Jul 19 17:02:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:02:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:02:02 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:02:02 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:03:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:03:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:03:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:03:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:04:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:04:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:04:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:04:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:05:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:05:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:05:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:05:01 serveur8 kernel: mdadm: sending ioctl 1261 to a partition!
Jul 19 17:06:52 serveur8 named[8515]: client 64.202.161.41#18478: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:06:53 serveur8 named[8515]: client 64.202.161.41#20542: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:06:56 serveur8 named[8515]: client 64.202.161.46#37120: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:06:56 serveur8 named[8515]: client 64.202.161.41#4905: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:01 serveur8 named[8515]: client 64.202.161.46#44078: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:01 serveur8 named[8515]: client 74.125.189.20#32895: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 17:07:01 serveur8 named[8515]: client 64.202.161.46#65349: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:04 serveur8 named[8515]: client 72.167.234.72#62955: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:04 serveur8 named[8515]: client 64.202.161.41#59740: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:04 serveur8 named[8515]: client 72.167.234.71#61830: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:04 serveur8 named[8515]: client 64.202.161.46#5410: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:05 serveur8 named[8515]: client 72.167.234.83#58021: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:05 serveur8 named[8515]: client 74.125.189.20#38495: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 17:07:08 serveur8 named[8515]: client 72.167.234.93#60519: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:09 serveur8 named[8515]: client 64.202.161.41#48997: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:10 serveur8 named[8515]: client 64.202.161.41#59638: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:11 serveur8 named[8515]: client 64.202.161.41#15173: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:11 serveur8 named[8515]: client 64.202.161.46#7873: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:13 serveur8 named[8515]: client 64.202.161.44#37017: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:13 serveur8 named[8515]: client 64.202.161.41#65327: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:14 serveur8 named[8515]: client 72.167.234.83#58859: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:16 serveur8 named[8515]: client 72.167.234.72#64634: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:17 serveur8 named[8515]: client 72.167.234.93#60212: query (cache) 'europa-ptp.com/A/IN' denied
Jul 19 17:07:29 serveur8 named[8515]: client 74.125.189.19#56913: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 17:07:31 serveur8 named[8515]: client 74.125.189.22#45585: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 17:07:31 serveur8 named[8515]: client 74.125.189.16#64149: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 17:07:31 serveur8 named[8515]: client 74.125.189.17#45084: query (cache) 'europa-ptp.com/NS/IN' denied
Jul 19 17:09:33 serveur8 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=70:54:d2:19:c4:c6:00:09:12:4a:2a:c2:08:00 SRC=85.154.245.155 DST=91.121.221.15 LEN=450 TOS=0x00 PREC=0x00 TTL=117 ID=28370 DF PROTO=UDP SPT=64489 DPT=80 LEN=430 
Jul 19 17:12:24 serveur8 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=70:54:d2:19:c4:c6:00:09:12:4a:2a:c2:08:00 SRC=173.242.116.191 DST=91.121.221.15 LEN=29 TOS=0x00 PREC=0x00 TTL=55 ID=0 DF PROTO=UDP SPT=39183 DPT=19 LEN=9 
Jul 19 17:13:56 serveur8 named[8515]: client 74.125.17.150#61783: query (cache) 'homactivity.eu/AAAA/IN' denied

Thanks for your help
 
Please check if you get any output of:
Code: 
grep MaxClients /var/log/httpd/error_log
grep MaxRequestWorkers /var/log/httpd/error_log
 
Last edited:
[Fri Jul 19 16:32:57 2013] [warn] Rejected, too many connections in READ state from 189.237.143.75
[Fri Jul 19 16:32:58 2013] [warn] Rejected, too many connections in READ state from 189.237.143.75
Click to expand...
These "too many connections from" als could be cause by mod_antiloris protection. Do you have mod_antiloris active?
If not, I'm not sure but maybe mod-evasive could cause these notices too.

However normally, Apache should not stop working when blocking an slowloris attack.
 
Hi
Yes, I have mod_antiloris, as all other servers. But only one has a problem.
There is no overload, and all seems working.

/etc/init.d/httpd status return "httpd (pid xxxxx) is running..."
but websites do not answer, and I've just to restart httpd.
 
When httpd is "stopped" => /etc/init.d/httpd graceful does nothing

settings MPM :

<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 20
ServerLimit 1200
MaxClients 1199
MaxRequestsPerChild 10000
</IfModule>

It is the same settings on all servers.
There is no problem of MaxClients, memory, swap, overload... so no idea.
 
Yes, I have mod_antiloris, as all other servers. But only one has a problem.
There is no overload, and all seems working.
Click to expand...
Exactly that is the reason I think that mod_antiloris might be the problem.
You could try to see what's happening when disabling mod_antiloris and only use mod_evasive.
 
Thanks Richard.

I change settings on this server.

=> I use APC cache on this one, and I'd put 8Go of memory.
It's too much, and maybe not detected.

I decrease to 2Go. It seem to be ok now, no new problem.
 
You must log in or register to reply here.
Back
Top