server overloading and crashing often

[bensaait]

Hello,

We've been having a lot of trouble with our server lately, we have to reboot it every two hours or so. We've been searching the web to find an answer but can find anyone to give us a straight answer, everyone guesses at the problem and suggests some sollution that only seems to make the problem worse.

I've attached a file with excerpts from all logs.

Thanks,
Ben

 

[nobaloney]

Well here's my guess...

Bad memory? Bad drives? Hacked Kernel? Denial of Service Attack ?

I'd want to see the top just before the crash, but that's going to be hard to find.

Can you log in via ssh, watch for the load to go up, and when it does, see which services are at the top of the top list?

Jeff

 

[bensaait]

I'd be happy to check that, but how can I do that?
What commands do I need?
I'm using freebsd

 

[netswitch]

bad memory can cause a lot of issues...

memtest86 could help you but I think you have to boot on it...

 

[nobaloney]

I'd be happy to check that, but how can I do that?
What commands do I need?

Do you have shell access to the server?

If so, run:

top

and watch.

Jeff

 

[bensaait]

Ok, I booted it up when the server seemed fine and got this

last pid: 18135;  load averages:  0.07,  0.20,  0.21                                                                                 up 1+04:43:04  03:44:19
86 processes:  1 running, 85 sleeping
CPU states:  0.4% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.2% idle
Mem: 275M Active, 34M Inact, 97M Wired, 11M Cache, 57M Buf, 46M Free
Swap: 1024M Total, 47M Used, 977M Free, 4% Inuse

  PID USERNAME   PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
  654 mysql       96    0 64524K  4340K select  23:53  0.88%  0.88% mysqld
18041 apache      20    0 24596K 19912K lockf    0:02  0.68%  0.68% httpd
18053 apache       4    0 24760K 20112K sbwait   0:01  0.59%  0.59% httpd
18079 apache       4    0 24744K 20084K sbwait   0:01  0.39%  0.39% httpd

All seems ok, but I did find this weird

86 processes: 1 running, 85 sleeping

 

[bensaait]

There is A LOT of that 'sbwait' stuff in there...

 

[bensaait]

Here's a full list

18064 apache       4    0 24828K 20160K sbwait   0:01  0.00%  0.00% httpd
  636 root        20    0  5476K   548K kserel   0:01  0.00%  0.00% named
last pid: 18227;  load averages:  0.23,  0.20,  0.21                                                                                 up 1+04:48:56  03:50:11
105 processes: 1 running, 104 sleeping
CPU states: 28.4% user,  0.0% nice,  3.5% system,  1.6% interrupt, 66.5% idle
Mem: 261M Active, 90M Inact, 95M Wired, 16M Cache, 57M Buf, 1012K Free
Swap: 1024M Total, 71M Used, 953M Free, 6% Inuse

  PID USERNAME   PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
  654 mysql       96    0 64524K  4132K select  24:00  0.93%  0.93% mysqld
18112 apache      20    0 24520K  6700K lockf    0:01  0.83%  0.83% httpd
18113 apache      20    0 24728K 10852K lockf    0:03  0.59%  0.59% httpd
18057 apache      20    0 24876K  6596K lockf    0:03  0.59%  0.59% httpd
18144 apache      20    0 24624K 10700K lockf    0:01  0.59%  0.59% httpd
18044 apache       4    0 24616K 10676K sbwait   0:03  0.49%  0.49% httpd
18045 apache      96    0 24616K 10648K select   0:02  0.49%  0.49% httpd
18049 apache       4    0 24680K 10692K sbwait   0:01  0.44%  0.44% httpd
18148 apache      20    0 24716K 10864K lockf    0:01  0.29%  0.29% httpd
18136 apache      20    0 24704K 10856K lockf    0:01  0.29%  0.29% httpd
18050 apache      20    0 24856K  8652K lockf    0:03  0.15%  0.15% httpd
18043 apache       4    0 24560K  8144K sbwait   0:01  0.15%  0.15% httpd
18053 apache      20    0 24760K 10792K lockf    0:03  0.10%  0.10% httpd
  681 root        96    0 19276K  4796K select   0:16  0.00%  0.00% httpd
18055 apache       4    0 25548K 10680K sbwait   0:03  0.00%  0.00% httpd
18042 apache      20    0 24880K 10928K lockf    0:03  0.00%  0.00% httpd
18048 apache       4    0 24564K  8392K sbwait   0:03  0.00%  0.00% httpd
18059 apache       4    0 24872K 10872K sbwait   0:03  0.00%  0.00% httpd
18058 apache      20    0 24516K 10536K lockf    0:03  0.00%  0.00% httpd
18068 apache       4    0 24536K 10552K sbwait   0:02  0.00%  0.00% httpd
18047 apache       4    0 24684K 10684K sbwait   0:02  0.00%  0.00% httpd
18051 apache       4    0 24520K  8496K sbwait   0:02  0.00%  0.00% httpd
18041 apache       4    0 24800K    12K sbwait   0:02  0.00%  0.00% httpd
18080 apache       4    0 24672K 10752K sbwait   0:02  0.00%  0.00% httpd
18079 apache       4    0 24744K  9260K sbwait   0:02  0.00%  0.00% httpd
18064 apache      20    0 24828K  8672K lockf    0:02  0.00%  0.00% httpd
18060 apache       4    0 24928K 10880K sbwait   0:02  0.00%  0.00% httpd
18116 apache      20    0 24876K 11024K lockf    0:02  0.00%  0.00% httpd
18146 apache      20    0 24736K  8720K lockf    0:02  0.00%  0.00% httpd
18114 apache       4    0 24520K  2716K sbwait   0:02  0.00%  0.00% httpd
18056 apache       4    0 24820K 10800K sbwait   0:02  0.00%  0.00% httpd
18078 apache      20    0 24540K 10624K lockf    0:02  0.00%  0.00% httpd
18072 apache       4    0 24716K  2068K sbwait   0:02  0.00%  0.00% httpd
18066 apache       4    0 24724K    12K sbwait   0:02  0.00%  0.00% httpd
18065 apache      20    0 24712K  6256K lockf    0:02  0.00%  0.00% httpd
  636 root        20    0  5476K   532K kserel   0:02  0.00%  0.00% named
18054 apache       4    0 24612K  5904K sbwait   0:01  0.00%  0.00% httpd
18075 apache       4    0 24508K  3096K sbwait   0:01  0.00%  0.00% httpd
18061 apache       4    0 24724K    12K sbwait   0:01  0.00%  0.00% httpd
18062 apache       4    0 24872K  8348K sbwait   0:01  0.00%  0.00% httpd
18073 apache       4    0 24808K    12K sbwait   0:01  0.00%  0.00% httpd
18052 apache      20    0 24672K 10704K lockf    0:01  0.00%  0.00% httpd
18074 apache       4    0 24888K    12K sbwait   0:01  0.00%  0.00% httpd
18070 apache       4    0 24724K  3788K sbwait   0:01  0.00%  0.00% httpd
18069 apache       4    0 24756K 10764K sbwait   0:01  0.00%  0.00% httpd
18137 apache       4    0 24724K 10876K sbwait   0:01  0.00%  0.00% httpd
18076 apache       4    0 24720K  8820K sbwait   0:01  0.00%  0.00% httpd
18115 apache       4    0 24740K    12K sbwait   0:01  0.00%  0.00% httpd
18063 apache       4    0 24740K  2348K sbwait   0:01  0.00%  0.00% httpd
  278 root        96    0  1312K   172K select   0:01  0.00%  0.00% syslogd
18173 apache       4    0 24544K  9416K sbwait   0:01  0.00%  0.00% httpd

 

[nobaloney]

A sleeping process is one that has been swapped out of memory by the task scheduler. It may be a process that ended but didn't get removed from memory. It will eventually be removed from memory and is using almost no system resources.

Jeff