Server crashing multiple times daily, started 5 days ago - please help

gkane · Sep 2, 2007

My server had been running fine for the past few months until 5 days ago. Now all of a sudden, I'm getting crashes multiple times daily. I'm working with my hosting service to troubleshoot, but we haven't made any progress tracking down the cause of the crashes.

I'm not sure exactly what causes these crashes, but I am unable to get to the DirectAdmin control panel and unable to connect with SSH. So, something seems to be bringing down all means of connecting to the server through the network. My only remedy is a remote power cycle. Then the server will come back up and perform normally for 4-12 hours, until it crashes again.

I'm running CentOS on a Athlon64 X2 3800+ 1GB RAM with Apache 2.x/MySQL5.x/PHP4.x. My two man sites are small vBulletin forums with an average total of 60-100 visitors across both sites. As far as I can tell, I have not been getting any more traffic than usual. I have previously had much higher traffic spikes without any problem.

The first major crash spit out a lot of "mysql_connect() Too many connections," but I'm now thinking that was not the cause, but rather a side effect of some other problem on the server. Since the first crash, I haven't had as many mysql errors. Looking at the logs, the mysql errors seem to happen after other errors. Today's mysql error log, did not show any errors during the time of the most recent crash.

Things my host and I have tried:

1. raising MySQL and Apache connection limits, MaxClients, ServerLimit
2. disabling all cron jobs
3. testing memory
4. replacing memory
5. testing hard drive for errors

None of those things have stopped or made any change in the frequency of the crashes.

I am now on the verge of moving to a new server, but I would rather solve the problem on the current server if possible. I also don't want to transfer all my sites over to a new server, just to have the same crashes start happening again.

Below I will paste some relevant parts of logs from the most recent crash. Please have a look and let me know any and all steps I can take to narrow down the cause of these crashes and prevent them from happening. Also let me know if there are any other log files that would help you make a better diagnosis.

Thank you for your help.

(Logs posted in the following post)

gkane · Sep 2, 2007

error logs

From 21:00 on, my sites were offline.

/var/log/messages

Code:

Sep  2 21:00:09 server_name kernel: httpd[5027]: segfault at 0000007fbf3fff68 rip 000000313293f6de rsp 0000007fbf3ffe10 error 6
Sep  2 21:02:03 server_name kernel: httpd[5239]: segfault at 0000007fbf3ffff8 rip 000000313296954f rsp 0000007fbf3fffd0 error 6
Sep  2 21:02:57 server_name kernel: httpd[7405]: segfault at 0000007fbf3ffff8 rip 000000313296954f rsp 0000007fbf3fffd0 error 6
Sep  2 21:02:57 server_name kernel: httpd[7400]: segfault at 0000007fbf3ffff8 rip 000000313296954f rsp 0000007fbf3fffd0 error 6
Sep  2 21:02:57 server_name kernel: httpd[7403]: segfault at 0000007fbf3ffff8 rip 000000313296954f rsp 0000007fbf3fffd0 error 6
Sep  2 21:05:25 server_name kernel: httpd[7841]: segfault at 0000007fbf3ffff8 rip 000000313296954f rsp 0000007fbf3fffd0 error 6
Sep  2 21:05:28 server_name kernel: httpd[7783]: segfault at 0000007fbf3ffff8 rip 000000313296954f rsp 0000007fbf3fffd0 error 6
Sep  2 21:06:01 server_name kernel: httpd[7452]: segfault at 0000007fbf3ffff8 rip 000000313296954f rsp 0000007fbf3fffd0 error 6
Sep  2 21:06:07 server_name kernel: httpd[7439]: segfault at 0000007fbf3ffff8 rip 000000313296954f rsp 0000007fbf3fffd0 error 6
Sep  2 21:06:13 server_name kernel: httpd[7882]: segfault at 0000007fbf3ffff8 rip 000000313296954f rsp 0000007fbf3fffd0 error 6
Sep  2 21:06:31 server_name kernel: httpd[6675]: segfault at 0000007fbf3ffff8 rip 000000313296954f rsp 0000007fbf3fffd0 error 6
Sep  2 21:21:19 server_name kernel: oom-killer: gfp_mask=0x1d2
Sep  2 21:29:12 server_name kernel: Mem-info:
Sep  2 21:29:12 server_name kernel: Node 0 DMA per-cpu:
Sep  2 21:29:12 server_name kernel: cpu 0 hot: low 2, high 6, batch 1
Sep  2 21:29:12 server_name kernel: cpu 0 cold: low 0, high 2, batch 1
Sep  2 21:29:12 server_name kernel: cpu 1 hot: low 2, high 6, batch 1
Sep  2 21:29:12 server_name kernel: cpu 1 cold: low 0, high 2, batch 1
Sep  2 21:29:12 server_name kernel: Node 0 Normal per-cpu:
Sep  2 21:29:12 server_name kernel: cpu 0 hot: low 32, high 96, batch 16
Sep  2 21:29:12 server_name kernel: cpu 0 cold: low 0, high 32, batch 16
Sep  2 21:29:12 server_name kernel: cpu 1 hot: low 32, high 96, batch 16
Sep  2 21:29:12 server_name kernel: cpu 1 cold: low 0, high 32, batch 16
Sep  2 21:29:12 server_name kernel: Node 0 HighMem per-cpu: empty
Sep  2 21:34:10 server_name kernel: 
Sep  2 21:35:24 server_name kernel: Free pages:       12752kB (0kB HighMem)
Sep  2 21:35:53 server_name kernel: Active:154482 inactive:20771 dirty:0 writeback:0 unstable:0 free:3188 slab:13605 mapped:176511 pagetables:41736
Sep  2 21:36:13 server_name kernel: Node 0 DMA free:11856kB min:12kB low:24kB high:36kB active:0kB inactive:0kB present:16384kB pages_scanned:394 all_unreclaimable? yes
Sep  2 21:36:23 server_name kernel: protections[]: 0 0 0
Sep  2 21:36:27 server_name kernel: Node 0 Normal free:896kB min:988kB low:1976kB high:2964kB active:617928kB inactive:83084kB present:1015744kB pages_scanned:1487046 all_unreclaimable? yes
Sep  2 21:36:39 server_name kernel: protections[]: 0 0 0
Sep  2 21:36:40 server_name kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Sep  2 21:36:45 server_name kernel: protections[]: 0 0 0
Sep  2 22:03:47 server_name kernel: Node 0 DMA: 6*4kB 3*8kB 2*16kB 4*32kB 2*64kB 2*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11856kB
Sep  2 22:03:48 server_name kernel: Node 0 Normal: 14*4kB 23*8kB 39*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 896kB
Sep  2 22:03:48 server_name kernel: Node 0 HighMem: empty
Sep  2 22:03:48 server_name kernel: Swap cache: add 659100, delete 659096, find 82500/98950, race 0+47
Sep  2 22:03:48 server_name kernel: Free swap:            0kB
Sep  2 22:03:48 server_name kernel: 258032 pages of RAM
Sep  2 22:03:48 server_name kernel: 5466 reserved pages
Sep  2 22:03:48 server_name kernel: 1253313 pages shared
Sep  2 22:03:48 server_name kernel: 4 pages swap cached
Sep  2 22:03:48 server_name kernel: Out of Memory: Killed process 5048 (mysqld).
Sep  2 22:03:48 server_name kernel: oom-killer: gfp_mask=0x1d2
Sep  2 22:03:48 server_name kernel: Mem-info:
Sep  2 22:03:48 server_name kernel: Node 0 DMA per-cpu:
Sep  2 22:03:48 server_name kernel: cpu 0 hot: low 2, high 6, batch 1
Sep  2 22:03:48 server_name kernel: cpu 0 cold: low 0, high 2, batch 1
Sep  2 22:03:48 server_name kernel: cpu 1 hot: low 2, high 6, batch 1
Sep  2 22:03:48 server_name kernel: cpu 1 cold: low 0, high 2, batch 1
Sep  2 22:03:48 server_name kernel: Node 0 Normal per-cpu:
Sep  2 22:03:48 server_name kernel: cpu 0 hot: low 32, high 96, batch 16
Sep  2 22:03:48 server_name kernel: cpu 0 cold: low 0, high 32, batch 16
Sep  2 22:03:48 server_name kernel: cpu 1 hot: low 32, high 96, batch 16
Sep  2 22:03:48 server_name kernel: cpu 1 cold: low 0, high 32, batch 16
Sep  2 22:03:48 server_name kernel: Node 0 HighMem per-cpu: empty
Sep  2 22:03:48 server_name kernel: 
Sep  2 22:03:48 server_name kernel: Free pages:       12816kB (0kB HighMem)
Sep  2 22:03:48 server_name kernel: Active:69091 inactive:106054 dirty:0 writeback:0 unstable:0 free:3204 slab:13587 mapped:176510 pagetables:41745
Sep  2 22:03:48 server_name kernel: Node 0 DMA free:11856kB min:12kB low:24kB high:36kB active:0kB inactive:0kB present:16384kB pages_scanned:394 all_unreclaimable? yes
Sep  2 22:03:48 server_name kernel: protections[]: 0 0 0
Sep  2 22:03:48 server_name kernel: Node 0 Normal free:960kB min:988kB low:1976kB high:2964kB active:276364kB inactive:424216kB present:1015744kB pages_scanned:1272381 all_unreclaimable? yes
Sep  2 22:03:48 server_name kernel: protections[]: 0 0 0
Sep  2 22:03:48 server_name kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Sep  2 22:03:48 server_name kernel: protections[]: 0 0 0
Sep  2 22:03:48 server_name kernel: Node 0 DMA: 6*4kB 3*8kB 2*16kB 4*32kB 2*64kB 2*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11856kB
Sep  2 22:03:48 server_name kernel: Node 0 Normal: 0*4kB 32*8kB 40*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 960kB
Sep  2 22:03:48 server_name kernel: Node 0 HighMem: empty
Sep  2 22:03:48 server_name kernel: Swap cache: add 659247, delete 659243, find 82508/98974, race 0+47
Sep  2 22:03:48 server_name kernel: Free swap:            0kB
Sep  2 22:03:48 server_name kernel: 258032 pages of RAM
Sep  2 22:03:48 server_name kernel: 5466 reserved pages
Sep  2 22:03:48 server_name kernel: 1244037 pages shared
Sep  2 22:03:48 server_name kernel: 4 pages swap cached
Sep  2 22:03:48 server_name kernel: Out of Memory: Killed process 5052 (mysqld).
Sep  2 22:03:48 server_name kernel: oom-killer: gfp_mask=0x1d2
Sep  2 22:03:48 server_name kernel: Mem-info:
Sep  2 22:03:48 server_name kernel: Node 0 DMA per-cpu:
Sep  2 22:03:48 server_name kernel: cpu 0 hot: low 2, high 6, batch 1
Sep  2 22:03:48 server_name kernel: cpu 0 cold: low 0, high 2, batch 1
Sep  2 22:03:48 server_name kernel: cpu 1 hot: low 2, high 6, batch 1
Sep  2 22:03:48 server_name kernel: cpu 1 cold: low 0, high 2, batch 1
Sep  2 22:03:48 server_name kernel: Node 0 Normal per-cpu:
Sep  2 22:03:48 server_name kernel: cpu 0 hot: low 32, high 96, batch 16
Sep  2 22:03:48 server_name kernel: cpu 0 cold: low 0, high 32, batch 16
Sep  2 22:03:48 server_name kernel: cpu 1 hot: low 32, high 96, batch 16
Sep  2 22:03:48 server_name kernel: cpu 1 cold: low 0, high 32, batch 16
Sep  2 22:03:48 server_name kernel: Node 0 HighMem per-cpu: empty
Sep  2 22:03:48 server_name kernel: 
Sep  2 22:03:48 server_name kernel: Free pages:       12816kB (0kB HighMem)
Sep  2 22:03:48 server_name kernel: Active:88687 inactive:86490 dirty:0 writeback:0 unstable:0 free:3204 slab:13587 mapped:176510 pagetables:41745
Sep  2 22:03:48 server_name kernel: Node 0 DMA free:11856kB min:12kB low:24kB high:36kB active:0kB inactive:0kB present:16384kB pages_scanned:394 all_unreclaimable? yes
Sep  2 22:03:48 server_name kernel: protections[]: 0 0 0
Sep  2 22:03:48 server_name kernel: Node 0 Normal free:960kB min:988kB low:1976kB high:2964kB active:354748kB inactive:345960kB present:1015744kB pages_scanned:1358643 all_unreclaimable? yes
Sep  2 22:03:48 server_name kernel: protections[]: 0 0 0
Sep  2 22:03:48 server_name kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Sep  2 22:03:48 server_name kernel: protections[]: 0 0 0
Sep  2 22:03:48 server_name kernel: Node 0 DMA: 6*4kB 3*8kB 2*16kB 4*32kB 2*64kB 2*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11856kB
Sep  2 22:03:48 server_name kernel: Node 0 Normal: 0*4kB 32*8kB 40*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 960kB
Sep  2 22:03:48 server_name kernel: Node 0 HighMem: empty
Sep  2 22:03:48 server_name kernel: Swap cache: add 659247, delete 659243, find 82508/98974, race 0+47
Sep  2 22:03:48 server_name kernel: Free swap:            0kB
Sep  2 22:03:48 server_name kernel: 258032 pages of RAM
Sep  2 22:03:48 server_name kernel: 5466 reserved pages
Sep  2 22:03:48 server_name kernel: 1244011 pages shared
Sep  2 22:03:48 server_name kernel: 4 pages swap cached
Sep  2 22:03:48 server_name kernel: Out of Memory: Killed process 5061 (mysqld).
Sep  2 22:03:48 server_name kernel: oom-killer: gfp_mask=0xd2
Sep  2 22:03:48 server_name kernel: Mem-info:
Sep  2 22:03:48 server_name kernel: Node 0 DMA per-cpu:
Sep  2 22:03:48 server_name kernel: cpu 0 hot: low 2, high 6, batch 1
Sep  2 22:03:48 server_name kernel: cpu 0 cold: low 0, high 2, batch 1
Sep  2 22:03:48 server_name kernel: cpu 1 hot: low 2, high 6, batch 1
Sep  2 22:03:48 server_name kernel: cpu 1 cold: low 0, high 2, batch 1
Sep  2 22:03:48 server_name kernel: Node 0 Normal per-cpu:
Sep  2 22:03:48 server_name kernel: cpu 0 hot: low 32, high 96, batch 16
Sep  2 22:03:48 server_name kernel: cpu 0 cold: low 0, high 32, batch 16
Sep  2 22:03:48 server_name kernel: cpu 1 hot: low 32, high 96, batch 16
Sep  2 22:03:48 server_name kernel: cpu 1 cold: low 0, high 32, batch 16
Sep  2 22:03:48 server_name kernel: Node 0 HighMem per-cpu: empty
Sep  2 22:03:48 server_name kernel: 
Sep  2 22:03:48 server_name kernel: Free pages:       12816kB (0kB HighMem)
Sep  2 22:03:48 server_name kernel: Active:91249 inactive:83992 dirty:0 writeback:0 unstable:0 free:3204 slab:13588 mapped:176510 pagetables:41745
Sep  2 22:03:48 server_name kernel: Node 0 DMA free:11856kB min:12kB low:24kB high:36kB active:0kB inactive:0kB present:16384kB pages_scanned:394 all_unreclaimable? yes
Sep  2 22:03:48 server_name kernel: protections[]: 0 0 0
Sep  2 22:03:48 server_name kernel: Node 0 Normal free:960kB min:988kB low:1976kB high:2964kB active:365124kB inactive:335968kB present:1015744kB pages_scanned:1436721 all_unreclaimable? yes
Sep  2 22:03:48 server_name kernel: protections[]: 0 0 0
Sep  2 22:03:48 server_name kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Sep  2 22:03:48 server_name kernel: protections[]: 0 0 0
Sep  2 22:03:48 server_name kernel: Node 0 DMA: 6*4kB 3*8kB 2*16kB 4*32kB 2*64kB 2*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11856kB
Sep  2 22:03:48 server_name kernel: Node 0 Normal: 0*4kB 32*8kB 40*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 960kB
Sep  2 22:03:48 server_name kernel: Node 0 HighMem: empty
Sep  2 22:03:48 server_name kernel: Swap cache: add 659247, delete 659243, find 82508/98974, race 0+47
Sep  2 22:03:48 server_name kernel: Free swap:            0kB
Sep  2 22:03:48 server_name kernel: 258032 pages of RAM
Sep  2 22:03:48 server_name kernel: 5466 reserved pages
Sep  2 22:03:48 server_name kernel: 1243873 pages shared
Sep  2 22:03:48 server_name kernel: 4 pages swap cached
Sep  2 22:03:48 server_name kernel: Out of Memory: Killed process 5065 (mysqld).
Sep  2 22:03:48 server_name kernel: oom-killer: gfp_mask=0x1d2
Sep  2 22:03:48 server_name kernel: Mem-info:
Sep  2 22:03:48 server_name kernel: Node 0 DMA per-cpu:
Sep  2 22:03:48 server_name kernel: cpu 0 hot: low 2, high 6, batch 1
Sep  2 22:03:48 server_name kernel: cpu 0 cold: low 0, high 2, batch 1
Sep  2 22:03:48 server_name kernel: cpu 1 hot: low 2, high 6, batch 1
Sep  2 22:03:48 server_name kernel: cpu 1 cold: low 0, high 2, batch 1
Sep  2 22:03:48 server_name kernel: Node 0 Normal per-cpu:
Sep  2 22:03:48 server_name kernel: cpu 0 hot: low 32, high 96, batch 16
Sep  2 22:03:48 server_name kernel: cpu 0 cold: low 0, high 32, batch 16
Sep  2 22:03:48 server_name kernel: cpu 1 hot: low 32, high 96, batch 16
Sep  2 22:03:48 server_name kernel: cpu 1 cold: low 0, high 32, batch 16

/var/log/httpd/error_log

Code:

[Sun Sep 02 21:00:10 2007] [notice] child pid 5027 exit signal Segmentation fault (11)
[Sun Sep 02 21:01:59 2007] [notice] child pid 5239 exit signal Segmentation fault (11)
[Sun Sep 02 21:03:10 2007] [notice] child pid 7400 exit signal Segmentation fault (11)
[Sun Sep 02 21:03:25 2007] [notice] child pid 7403 exit signal Segmentation fault (11)
[Sun Sep 02 21:03:25 2007] [notice] child pid 7405 exit signal Segmentation fault (11)
[Sun Sep 02 21:05:24 2007] [notice] child pid 7841 exit signal Segmentation fault (11)
[Sun Sep 02 21:05:28 2007] [notice] child pid 7783 exit signal Segmentation fault (11)
[Sun Sep 02 21:06:04 2007] [notice] child pid 7452 exit signal Segmentation fault (11)
[Sun Sep 02 21:06:07 2007] [notice] child pid 7439 exit signal Segmentation fault (11)
[Sun Sep 02 21:06:13 2007] [notice] child pid 7882 exit signal Segmentation fault (11)
[Sun Sep 02 21:06:31 2007] [notice] child pid 6675 exit signal Segmentation fault (11)
[Sun Sep 02 21:33:10 2007] [error] server reached MaxClients setting, consider raising the MaxClients setting

joeker · Sep 5, 2007

Looks like a hardware problem

Server crashing multiple times daily, started 5 days ago - please help

gkane

Verified User

gkane

Verified User

joeker

Verified User