Server crashes / mem issues

marcovis

New member
Joined
Jul 9, 2014
Messages
2
Dear all,

I'm running a CentOS 6.5 server, with 2 GB RAM and DA. The server hosts less than 20 sites and those sites do not have many visitors.

Frequently, I'm experiencing server issues: server becomes unrepsonsive. When I look at the console I see some remarks from the OOM-killer ("sacrifice process or kill child"), but I cannot logon. I then need to reboot to get the server / sites back online.

This happens sometimes twice a week, sometimes there are intervals of two weeks or more. Sometimes the interval is a day.

Last time, I found in de DA error log the following (see below) - "FORK ERROR - You're probably out of memory". This seems to be in line with OOM-killer occuring.

The server has 2 GB of memory, it's on a VMWare and those 2 GB are dedicated. VMW host is on the limits of memory but not overbooked. There are some other servers on this VMW host and none of them is having simular issues.

What I've done so far is added 'vm.overcommit_memory = 2' and 'vm.overcommit_ratio = 50' to SYSCTL.conf.

Does anybody have tips / remarks / suggestions?

Kind regards,
Marco Visser

Code:
2015:03:15-21:26:52: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)
2015:03:19-20:31:38: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:31:48: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:31:58: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:32:16: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:32:26: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:32:36: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:33:11: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:33:16: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)
2015:03:19-20:33:21: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:33:29: Out of memory: cannot allocate space for: images/left-shadow.gif
2015:03:19-20:33:29: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)
2015:03:19-20:33:29: Error while sending ./data/skins/enhanced/images/nav-home-over.gif
2015:03:19-20:33:31: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:33:41: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:33:51: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:33:53: Out of memory: cannot allocate space for: IMG_SKIN_LEFT_SHADOW
2015:03:19-20:34:08: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:34:18: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:34:28: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:34:38: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:34:48: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:34:59: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:35:09: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:35:19: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:35:46: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:36:02: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:36:24: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:36:34: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:36:44: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:36:54: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:37:27: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:37:37: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:37:47: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:37:54: Unable to read ./data/admin/login.hist: Unable to get Lock on file:<br>
file is locked by another process<br>

2015:03:19-20:37:56: Unable to read ./data/admin/login.hist: Unable to get Lock on file:<br>
file is locked by another process<br>

2015:03:19-20:37:57: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:38:41: Out of memory: cannot allocate space for: Unable to add access host:
2015:03:19-20:38:45: Out of memory: cannot allocate space for: images/header.gif
2015:03:19-20:38:45: Out of memory: cannot allocate space for: IMG_SKIN_LEFT_SHADOW
2015:03:19-20:38:50: Out of memory: cannot allocate space for: images/header.gif
2015:03:19-20:38:54: Out of memory: cannot allocate space for: new=yes&type=message
2015:03:19-20:38:56: *** Segmentation fault *** getlock(./data/users/marcovis/login.hist) : finished : User: marcovis : 83.162.202.245 : / : (null) : (null)
2015:03:19-20:40:06: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:40:16: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:40:27: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:48:39: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:48:49: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:49:00: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:49:10: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:49:20: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:49:37: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:49:58: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:50:09: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:50:21: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:50:48: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:50:58: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:51:07: Out of memory: cannot allocate space for: 161
2015:03:19-20:51:09: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:51:12: Out of memory: cannot allocate space for: new=yes&type=message
2015:03:19-20:51:12: *** Segmentation fault *** getlock(./data/users/marcovis/login.hist) : finished : User: marcovis : 83.162.253.173 : / : (null) : (null)
2015:03:19-20:51:17: Out of memory: cannot allocate space for: HTM_FOOTER
2015:03:19-20:51:21: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:51:22: Out of memory: cannot allocate space for: 169
2015:03:19-20:51:31: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:52:08: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:52:18: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:52:30: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:52:43: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:53:02: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:53:17: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:53:27: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:53:37: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:53:50: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:54:08: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:54:18: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:54:28: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:55:56: Out of memory: cannot allocate space for: application/vnd.ms-wpl
2015:03:19-20:55:59: Out of memory: cannot allocate space for: application/vnd.ms-powerpoint.addin.macroEnabled.12
2015:03:19-20:56:00: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:56:10: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:56:20: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:56:32: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:56:42: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:56:52: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:57:05: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:57:10: *** Segmentation fault *** getlock(./data/users/marcovis/login.hist) : finished : User: marcovis : 83.162.253.173 : /none : (null) : (null)
2015:03:19-20:57:10: *** Segmentation fault *** getlock(./data/users/marcovis/login.hist) : finished : User: marcovis : 83.162.253.173 : / : (null) : (null)
2015:03:19-20:57:10: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)
2015:03:19-20:57:10: Socket write error: fd is connected to a pipe or socket whose reading end is closed.  When this  happens the writing process will also receive a SIG_PIPE signal.  (Thus, the write return value is seen only if  the program catches, blocks or ignores this signal.)
2015:03:19-20:57:15: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:57:25: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:57:35: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:57:46: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:57:59: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:58:09: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:58:22: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:59:02: Out of memory: cannot allocate space for: application/x-bcpio
2015:03:19-20:59:02: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:59:12: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
2015:03:19-20:59:22: FORK ERROR - You're probably out of memory since fork() returned -1. Sleeping for 10 seconds.
 
Hello,

1. use nginx+apache
2. set lower max allowed processes/threads in apache, mysql, dovecot, exim
3. disable if you don't use InnoDB in MySQL
4. optimize mysql to use less RAM
 
Back
Top