apache stop Suddenly

asishlla · Sep 20, 2010

hello every one
why apache stop some times?
and i need to make reboot to start it
thanks

R1Lover · Sep 20, 2010

check the error logs

daveyw · Sep 20, 2010

Restarting apache is enough, no need to reboot the whole server.
The reason why something stopped/crashed, you can find in the error logs.

asishlla · Sep 20, 2010

this is the httpd error logs

Code:

shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
[Mon Sep 20 08:26:52 2010] [error] [client 87.117.197.184] [11229] cannot create mutex /var/tmp/clamav.lock
[Mon Sep 20 08:26:52 2010] [error] [client 87.117.197.184] [11229] cannot create mutex /var/tmp/clamav.lock
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clama[Mon Sep 20 08:27:23 2010] [error] [client 87.117.197.184] [11598] cannot create mutex /var/tmp/clamav.lock
[Mon Sep 20 08:27:23 2010] [error] [client 87.117.197.184] [11598] cannot create mutex /var/tmp/clamav.lock

and this is the directadmin error log

Code:

2010:09:19-09:49:32: ashiqqa.org already exists on the server. It will not be restored.
2010:09:19-09:49:32: Unable to stat /home/nrgela/backups/backup/nrgela.com/email/passwd as root.  File probably doesn't exist
2010:09:19-09:49:33: Unable to stat /home/nrgela/backups/backup/nrgela.com/email/aliases as root.  File probably doesn't exist
2010:09:19-09:49:33: Unable to stat /home/nrgela/backups/backup/nrgela.com/email/autoresponder.conf as root.  File probably doesn't exist
2010:09:19-09:49:33: Unable to stat /home/nrgela/backups/backup/nrgela.com/email/vacation.conf as root.  File probably doesn't exist
2010:09:19-09:49:33: Can't add user nrgela_ashiqqa to nrgela_ashiqqa: 
Password must have numbers
<br>
<br>


2010:09:19-09:50:33: The username hasn't been set.  Won't run the script without dropping privileges
2010:09:19-09:51:09: The username hasn't been set.  Won't run the script without dropping privileges
2010:09:19-09:51:29: ashiqqa.org already exists on the server. It will not be restored.
2010:09:19-09:51:29: Unable to stat /home/nrgela/backups/backup/nrgela.com/email/passwd as root.  File probably doesn't exist
2010:09:19-09:51:29: Unable to stat /home/nrgela/backups/backup/nrgela.com/email/aliases as root.  File probably doesn't exist
2010:09:19-09:51:29: Unable to stat /home/nrgela/backups/backup/nrgela.com/email/autoresponder.conf as root.  File probably doesn't exist
2010:09:19-09:51:29: Unable to stat /home/nrgela/backups/backup/nrgela.com/email/vacation.conf as root.  File probably doesn't exist
2010:09:19-09:51:29: Can't add user nrgela_ashiqqa to nrgela_ashiqqa: 
Password must have numbers
<br>
<br>


2010:09:19-09:58:21: Referer host does not match sn141w.snt141.mail.live.com != 87.117.197.184
2010:09:19-09:58:22: Referer check failed for 78.89.18.219
2010:09:19-10:05:45: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)
2010:09:19-10:05:45: Error while sending ./data/skins/enhanced/images/header.gif
2010:09:19-11:17:39: Referer host does not match us.mg3.mail.yahoo.com != 87.117.197.184
2010:09:19-11:17:39: Referer check failed for 109.107.233.31
2010:09:19-11:18:34: File ./data/users/anati/user.usage has been written to after this process read it.  Not going to overwrite it.
2010:09:19-13:34:01: Socket write error: Unknown error
2010:09:19-13:34:01: Error while sending ./data/skins/enhanced/images/header.gif
2010:09:19-13:52:15: Timeout from from 41.249.20.64 : last flagged: Request::readAndProcess(*skt, 41.249.20.64, 41.249.20.64)
2010:09:19-13:53:15: Timeout from from 41.249.20.64 : last flagged: Request::readAndProcess(*skt, 41.249.20.64, 41.249.20.64)
2010:09:19-13:54:26: Timeout from from 41.249.20.64 : last flagged: Request::readAndProcess(*skt, 41.249.20.64, 41.249.20.64)
2010:09:19-13:54:26: Timeout from from 41.249.20.64 : last flagged: Request::readAndProcess(*skt, 41.249.20.64, 41.249.20.64)
2010:09:19-13:54:37: Timeout from from 41.249.20.64 : last flagged: Request::readAndProcess(*skt, 41.249.20.64, 41.249.20.64)
2010:09:19-13:55:26: Timeout from from 41.249.20.64 : last flagged: Request::readAndProcess(*skt, 41.249.20.64, 41.249.20.64)
2010:09:19-20:24:01: Auth::passValid: unable to get user_info for areawebh
2010:09:19-20:25:06: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)
2010:09:19-22:56:57: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)
2010:09:19-23:05:48: Error changing ftp password for amani: 
Password must have numbers
<br>
<br>

2010:09:20-00:41:32: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)
2010:09:20-07:28:31: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)
2010:09:20-07:44:31: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)

LawsHosting · Sep 20, 2010

As we have found out yesterday, it was a hack, which you fixed. However, dispite you fixing it, it won't stop the scans for the hack. My thoughts would be that they're hitting your apache so hard its coming under strain.

To find out, note the time in the logs:

Code:

cat /var/log/httpd/domains/*.log | grep .txt??

if its a lot per ~10 seconds then this could be why apache is dying.

What are your MaxClients & MaxRequestsPerChild in your httpd-mpm.conf set to?

asishlla · Sep 20, 2010

cat /var/log/httpd/domains/*.log | grep .txt??
in ssh = one second > no result > new line

What are your MaxClients & MaxRequestsPerChild in your httpd-mpm.conf set to?
how could i know?

LawsHosting · Sep 20, 2010

asishlla said:
What are your MaxClients & MaxRequestsPerChild in your httpd-mpm.conf set to? how could i know?

nano /etc/httpd/conf/extra/httpd-mpm.conf

asishlla · Sep 20, 2010

Code:

#
# Server-Pool Management (MPM specific)
# 

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
#
# Note that this is the default PidFile for most MPMs.
#
<IfModule !mpm_netware_module>
    PidFile "/var/run/httpd.pid"
</IfModule>

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
<IfModule !mpm_winnt_module>
<IfModule !mpm_netware_module>
LockFile /var/logs/accept.lock
</IfModule>
</IfModule>

#
# Only one of the below sections will be relevant on your
# installed httpd.  Use "apachectl -l" to find out the
# active mpm.
#

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    ServerLimit            450
    MaxClients          450
    MaxRequestsPerChild   10000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    StartServers          2
    ServerLimit         450
    MaxClients          450
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadsPerChild      25
    MaxRequestsPerChild   10000
</IfModule>

# BeOS MPM
# StartThreads: how many threads do we initially spawn?
# MaxClients:   max number of threads we can have (1 thread == 1 client)
# MaxRequestsPerThread: maximum number of requests each thread will process
<IfModule mpm_beos_module>
    StartThreads            10
    MaxClients              50
    MaxRequestsPerThread 10000
</IfModule>

# NetWare MPM
# ThreadStackSize: Stack size allocated for each worker thread
# StartThreads: Number of worker threads launched at server startup
# MinSpareThreads: Minimum number of idle threads, to handle request spikes
# MaxSpareThreads: Maximum number of idle threads
# MaxThreads: Maximum number of worker threads alive at the same time
# MaxRequestsPerChild: Maximum  number of requests a thread serves. It is 
#                      recommended that the default value of 0 be set for this
#                      directive on NetWare.  This will allow the thread to 
#                      continue to service requests indefinitely.                          
<IfModule mpm_netware_module>
    ThreadStackSize      65536
    StartThreads           250
    MinSpareThreads         25
    MaxSpareThreads        250
    MaxThreads            1000
    MaxRequestsPerChild      10000
    MaxMemFree             100
</IfModule>

# OS/2 MPM
# StartServers: Number of server processes to maintain
# MinSpareThreads: Minimum number of idle threads per process, 
#                  to handle request spikes
# MaxSpareThreads: Maximum number of idle threads per process
# MaxRequestsPerChild: Maximum number of connections per server process
<IfModule mpm_mpmt_os2_module>
    StartServers           2
    MinSpareThreads        5
    MaxSpareThreads       10
    MaxRequestsPerChild    10000
</IfModule>

LawsHosting · Sep 20, 2010

MaxClients 450
MaxRequestsPerChild 10000

Looks okay to me......

What are the server specs, or the log at the time of the crash? If any.

asishlla · Sep 20, 2010

crash happen at morninng yestrday and today

asishlla · Sep 20, 2010

daveyw said:
Restarting apache is enough, no need to reboot the whole server.
The reason why something stopped/crashed, you can find in the error logs.

thank u daveyw
but when i try to restart it it work for second and stop again because of that i do reboot

LawsHosting · Sep 20, 2010

Does the apache error_log indicate anything abnormal around that time? And when it crashes, does it come back up itself?

Code:

cat /var/log/httpd/error_log | grep <time>

<time> - in 24hours.

asishlla · Sep 20, 2010

Peter Laws said:
Does the apache error_log indicate anything abnormal around that time? And when it crashes, does it come back up itself?

Code:

cat /var/log/httpd/error_log | grep <time>

<time> - in 24hours.

there is photo attached man

LawsHosting · Sep 20, 2010

asishlla said:
there is photo attached man

I meant <time> as in you add the time

Try this, if you know it crashes between 12 and 1pm:

Code:

cat /var/log/httpd/error_log | awk '/12:00:00/{p=1} /13:00:00/{p=0} p'

I'm no bash king though.

asishlla · Sep 20, 2010

hello friend
now 7.00 evening another crush!
it make me sick

LawsHosting · Sep 20, 2010

So look at the logs for around 7pm......... Or tail the logs so you'll see the log update in realtime:

Code:

tail -f /var/log/httpd/error_log

when it crashes, you'll see why instantly.

Tap Ctrl & X to break out the tail command when finished.

Or failing that, employ someone to look into the causes.....

asishlla · Sep 21, 2010

thank you peter
every thing now work fine i think! but look to httpd error log:
how can i fix it?

Code:

shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clamav.shm: statistics will not be available
shmem.c:35: cannot create shared memory /var/tmp/clama[Tue Sep 21 13:33:30 2010] [error] [client 87.117.197.184] [1218] cannot create mutex /var/tmp/clamav.lock
[Tue Sep 21 13:33:30 2010] [error] [client 87.117.197.184] [1218] cannot create mutex /var/tmp/clamav.lock

LawsHosting · Sep 21, 2010

It could mean you are running out of memory when clamav is running.

On a side note, do u use custombuild? If so, did you secure your php.ini file?

asishlla · Sep 22, 2010

I want to thank Peter from the depths of my heart
You are an example of professionalism and good handling and help!
I am sure that your customers are very happy because they are in competent hands,thank you !

apache stop Suddenly

asishlla

Verified User

R1Lover

Verified User

daveyw

Verified User

asishlla

Verified User

LawsHosting

Verified User

asishlla

Verified User

LawsHosting

Verified User

asishlla

Verified User

LawsHosting

Verified User

asishlla

Verified User

asishlla

Verified User

LawsHosting

Verified User

asishlla

Verified User

Attachments

LawsHosting

Verified User

asishlla

Verified User

LawsHosting

Verified User

asishlla

Verified User

LawsHosting

Verified User

asishlla

Verified User