High server load

nealdxmhost

Verified User
Joined
Jan 1, 2009
Messages
232
Location
Los Angeles CA
A server I manage was hit with an incredibly high load this morning that nearly brought it to its knees.

Details on the server:
Dell Poweredge R710 with 8 300GB SAS drives in a RAID10 configuration to include FOUR hot spares, 24 GB RAM, 2 E5620 CPU
OS CENTOS 6.6
DA Custombuild 2.0
CSF and BFM installed
Two wordpress sites running on it along with two installations of WHMCS
Here is what I got this morning:

Code:
This is an automated message notifying you that the 5 minute load average on your system is 117.1.
This has exceeded the 10 threshold.

One Minute      - 353.8
Five Minutes    - 117.1
Fifteen Minutes - 41.99

top - 08:26:43 up 21 days,  3:38,  0 users,  load average: 353.80, 117.10, 41.99
Tasks: 786 total,  13 running, 773 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  49409432k total,  9380944k used, 40028488k free,   408564k buffers
Swap: 24772604k total,        0k used, 24772604k free,  1219588k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
  13 root      20   0     0    0    0 R 100.0  0.0   4:46.82 [ksoftirqd/2]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
  45 root      20   0     0    0    0 R 100.0  0.0   6:51.79 [ksoftirqd/10]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  53 root      20   0     0    0    0 R 100.0  0.0   4:45.32 [ksoftirqd/12]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  65 root      20   0     0    0    0 R 100.0  0.0   6:48.06 [ksoftirqd/15]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  17 root      20   0     0    0    0 R 99.3  0.0   4:45.70 [ksoftirqd/3]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
  61 root      20   0     0    0    0 R 99.3  0.0   4:45.47 [ksoftirqd/14]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
  67 root      20   0     0    0    0 R 99.3  0.0   4:05.24 [events/0]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
25913 root      20   0 15560 1684  828 R  3.8  0.0   0:00.03 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
   1 root      20   0 19364 1548 1232 S  0.0  0.0   0:04.79 /sbin/init                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
   2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 [kthreadd]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
   3 root      RT   0     0    0    0 S  0.0  0.0   0:00.14 [migration/0]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
   4 root      20   0     0    0    0 R  0.0  0.0   3:43.25 [ksoftirqd/0]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
   5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 [stopper/0]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
   6 root      RT   0     0    0    0 S  0.0  0.0   0:02.08 [watchdog/0]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
   7 root      RT   0     0    0    0 S  0.0  0.0   0:00.02 [migration/1]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
   8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 [stopper/1]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
   9 root      20   0     0    0    0 S  0.0  0.0   0:04.07 [ksoftirqd/1]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
  10 root      RT   0     0    0    0 S  0.0  0.0   0:01.43 [watchdog/1]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  11 root      RT   0     0    0    0 S  0.0  0.0   0:00.53 [migration/2]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
  12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 [stopper/2]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
  14 root      RT   0     0    0    0 S  0.0  0.0   2:02.74 [watchdog/2]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  15 root      RT   0     0    0    0 S  0.0  0.0   0:00.07 [migration/3]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
  16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 [stopper/3]
 
I'm no hardware guy, but I know ksoftirqd is running to queue a high irq load.
Question is where the high load of interrupt request are or were coming from. Maybe a defective raid controller?
 
Back
Top