Lots of overload messages in one moment

Muniek · Oct 23, 2013

Hello. I have a few dedicated servers. Everyone is configured the same, at last it should be the same, because i have problems with one of them. On every server i have DirectAdmin and one of them few weeks ago starts acting weird. I'm reciving a lot of overload messages (value over 50.0 per 15 minutes). I set checking it every hour, but it's still sending me about 15 messages one minute with the same 'top' log. On rest servers it runs perfectly. As you can see in logs some of the processes using 9999% od CPU. Can somebody tell me something about this? Mayby someone will know how to fix it, because it is really annoying - i can't even log via SSH to root while server is overloaded.

Code:

000000426	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000425	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000424	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000423	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000422	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000421	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000420	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000419	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000418	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000417	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000416	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000415	Warning: The system load average is 92.56	Dzisiaj o 15:30	
000000414	Warning: The system load average is 92.56	Dzisiaj o 15:30

Code:

This is an automated message notifying you that the 15 minute load average on your system is 92.56.
This has exceeded the 50 threshold.

One Minute      - 170.39
Five Minutes    - 155.16
Fifteen Minutes - 92.56

top - 15:30:22 up 3 days,  5:38,  0 users,  load average: 170.39, 155.16, 92.56
Tasks: 473 total,  32 running, 417 sleeping,   0 stopped,  24 zombie
Cpu(s):  0.4%us,  1.7%sy, 14.7%ni, 82.9%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  66008460k total, 65684176k used,   324284k free,   277360k buffers
Swap:  8386552k total,    17904k used,  8368648k free, 36046528k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
11622 root      20   0 41120 3032 2432 S 9999  0.0 600517:28 /usr/local/directadmin/dataskq                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
11872 root      20   0 19380 1580  916 S 9999  0.0 300258:44 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
23632 tcagame   39  19 4039m 2.6g  12m S 9999  4.2  5901654h java -Xmx3072M -Xms3072M -Djline.terminal=jline.UnsupportedTerminal -jar craftbukkit.jar --nojline nogui                                                                                                                                                                                                                                                                                                                                                                                                                           
29245 tcagame   39  19 1006m 592m  12m S 9999  0.9  51675,21 java -Xmx512M -Xms512M -Djline.terminal=jline.UnsupportedTerminal -jar craftbukkit.jar --nojline nogui                                                                                                                                                                                                                                                                                                                                                                                                                             
11772 root      20   0 19380 1596  920 R    3  0.0   0:00.02 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
11790 root      20   0     0    0    0 Z    3  0.0   0:00.02 [top] <defunct>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
31144 tcagame   39  19 1174m 472m  12m S    3  0.7  7686304h java -Xmx768M -Xms768M -Djline.terminal=jline.UnsupportedTerminal -jar craftbukkit.jar --nojline nogui                                                                                                                                                                                                                                                                                                                                                                                                                             
11774 root      20   0 19388 1604  932 R    1  0.0   0:00.01 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
11777 root      20   0 19380 1596  920 R    1  0.0   0:00.01 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
11781 root      20   0 19380 1596  920 R    1  0.0   0:00.01 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
11788 root      20   0 19380 1596  920 R    1  0.0   0:00.01 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
11799 root      20   0 19380 1596  920 R    1  0.0   0:00.01 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
11869 root      20   0 19380 1588  916 S    1  0.0   0:00.01 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
   1 root      20   0  8396  696  668 S    0  0.0  5126869h init [2]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
   2 root      20   0     0    0    0 S    0  0.0   0:00.00 [kthreadd]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
   3 root      20   0     0    0    0 R    0  0.0  5119411h [ksoftirqd/0]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
   5 root      20   0     0    0    0 S    0  0.0 600517:29 [kworker/u:0]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
   6 root      RT   0     0    0    0 S    0  0.0 304614:03 [migration/0]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
   7 root      RT   0     0    0    0 S    0  0.0   0:00.00 [migration/1]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
   8 root      20   0     0    0    0 S    0  0.0  80068,59 [kworker/1:0]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
   9 root      20   0     0    0    0 S    0  0.0   0:02.20 [ksoftirqd/1]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
  11 root      RT   0     0    0    0 S    0  0.0   0:00.00 [migration/2]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
  12 root      20   0     0    0    0 S    0  0.0  35030,11 [kworker/2:0]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      


================================
Automated Message Generated by DirectAdmin

nobaloney · Oct 23, 2013

Check your task queue. restart the dataskq.

Jeff

Muniek · Oct 23, 2013

I used this: http://help.directadmin.com/item.php?id=107

Code:

root@sX:~# cat /usr/local/directadmin/data/task.queue
cat: /usr/local/directadmin/data/task.queue: Nie ma takiego pliku ani katalogu (File doesn't exist)

It runs every minute:

Code:

(...)
Oct 24 01:01:01 sX /USR/SBIN/CRON[15230]: (root) CMD (/usr/local/directadmin/dataskq)
Oct 24 01:02:01 sX /USR/SBIN/CRON[16846]: (root) CMD (/usr/local/directadmin/dataskq)
Oct 24 01:03:01 sX /USR/SBIN/CRON[18353]: (root) CMD (/usr/local/directadmin/dataskq)
Oct 24 01:04:01 sX /USR/SBIN/CRON[19773]: (root) CMD (/usr/local/directadmin/dataskq)
Oct 24 01:05:01 sX /USR/SBIN/CRON[21161]: (root) CMD (/usr/local/directadmin/dataskq)
Oct 24 01:06:01 sX /USR/SBIN/CRON[22880]: (root) CMD (/usr/local/directadmin/dataskq)

Cron is running.

Code:

root@sX:~# ps ax | grep cron
 3804 ?        Ss   21178179:40 /usr/sbin/cron
25045 pts/2    S+     0:00 grep cron

Muniek · Oct 23, 2013

Hi! Today i have another 22 messages with the same time and content.

Code:

This is an automated message notifying you that the 15 minute load average on your system is 120.32.
This has exceeded the 50 threshold.

One Minute      - 157.39
Five Minutes    - 151.83
Fifteen Minutes - 120.32

top - 07:37:41 up 3 days, 21:45,  0 users,  load average: 181.34, 174.85, 149.32
Tasks: 461 total,  13 running, 448 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4%us,  1.8%sy, 16.7%ni, 80.7%id,  0.3%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  66008460k total, 62857220k used,  3151240k free,   277920k buffers
Swap:  8386552k total,    24224k used,  8362328k free, 35683956k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
27269 root      20   0 19380 1592  932 R 9999  0.0  5124095h /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1118 root      20   0 41120 3032 2432 S 9999  0.0 300258:44 /usr/local/directadmin/dataskq                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
1153 root      20   0 14092 1456 1244 S 9999  0.0 300258:44 sh /root/cpu_stats.sh                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
25844 root      20   0 19240 1468  916 R 9999  0.0  20017,14 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1587 root      20   0  6104  692  560 S 9999  0.0 300258:44 vmstat 1 2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
1616 root      20   0  6788  552  476 S 9999  0.0 300258:44 sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1651 mail      20   0 43140 3348 2152 S 9999  0.0 300258:44 /usr/sbin/exim -Mc 1VZDc5-0000QX-ED                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1659 mail      20   0 43140 2360 1180 S 9999  0.0 300258:44 /usr/sbin/exim -Mc 1VZDc5-0000QK-Ch                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
26897 root      20   0 19372 1536  920 R 9999  0.0 300258:44 /usr/bin/top -c -b -n 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
9961 tcagame   39  19  791m 526m  12m S 9999  0.8  7769552h java -Xmx384M -Xms384M -Djline.terminal=jline.UnsupportedTerminal -jar craftbukkit.jar --nojline nogui                                                                                                                                                                                                                                                                                                                                                                                                                             
1158 root      20   0 17532 1352 1128 S 9999  0.0  5119411h sh -c /usr/bin/top -c -b -n 1  | /usr/bin/head -n 30 2>&1                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
1160 root      20   0  3924  292  232 S 9999  0.0  5119411h /usr/bin/head -n 30                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1420 root      20   0  6788  552  476 S 9999  0.0  5119411h sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1424 root      20   0  6788  552  476 S 9999  0.0  5119411h sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1456 root      20   0  6788  552  476 S 9999  0.0  5119411h sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1478 root      20   0  6788  552  476 S 9999  0.0  5119411h sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1524 root      20   0  6104  692  560 S 9999  0.0  5119411h vmstat 1 2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
1538 root      20   0  6788  552  476 S 9999  0.0  5119411h sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1556 root      20   0  6788  552  476 S 9999  0.0  5119411h sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1570 root      20   0  6788  552  476 S 9999  0.0  5119411h sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1609 root      20   0  6788  552  476 S 9999  0.0  5119411h sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1610 root      20   0  6788  552  476 S 9999  0.0  5119411h sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
1615 root      20   0  6788  552  476 S 9999  0.0  5119411h sleep 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            


================================
Automated Message Generated by DirectAdmin

nobaloney · Oct 24, 2013

Does your server really have 66 GB of memory? How many processor cores? Is your server still running well with such a high server load?

Jeff

Muniek · Oct 24, 2013

Yes, i have 64 GB DDR3 ECC with CPU Intel Xeon E5-1650 (6 cores / 12 Threads) 3.2GHz (3.8GHz Turbo Boost). Every server has the same hardware. Server runs normally with load about 30-40. Today i saw something new - i could ping server, i can access sX.domain.com (Apache is functioning normally) but i couldn't access SSH or Directadmin on port 2222.

nobaloney · Oct 24, 2013

Unfortunately I don't have any answers except to wonder what's in your task.queue that it would show up long enough to be in the top command.

Jeff

Muniek · Oct 25, 2013

Is there any way to compare configuration of 2 dedicated servers? Or can i reinstall whole directadmin?

zEitEr · Oct 27, 2013

compare configuration of 2 dedicated servers

If you want to get benchmark of your server, you can try this: http://code.google.com/p/byte-unixbench/

Or can i reinstall whole directadmin?

It's not recommended to do that. If you really(?) need to re-install Directadmin you'd better format your HDDs and re-install OS, and only then you install Directadmin.

Please, note, if you need somebody to investigate the issue with your server, please feel free to contact those of us here who answers in your thread for a quote. I'd be glad to assist you.

Muniek · Nov 13, 2013

Hello. I'm still having problems but i found this in my /var/log/syslog:

Code:

(...)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19139]: (root) CMD (sh /root/cpu_stats.sh)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19138]: (root) CMD (/usr/local/rtm/bin/rtm 49 > /dev/null 2> /dev/null)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19140]: (root) CMD (/usr/local/directadmin/dataskq)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19148]: (root) CMD (sh /root/cpu_stats.sh)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19153]: (root) CMD (/usr/local/rtm/bin/rtm 49 > /dev/null 2> /dev/null)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19154]: (root) CMD (sh /root/cpu_stats.sh)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19458]: (root) CMD (sh /root/cpu_stats.sh)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19464]: (root) CMD (/usr/local/directadmin/dataskq)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19462]: (root) CMD (   sh /root/disk_stats.sh)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19461]: (root) CMD (/usr/local/directadmin/dataskq)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19475]: (root) CMD (sh /root/cpu_stats.sh)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19476]: (root) CMD (/usr/local/directadmin/dataskq)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19481]: (root) CMD (/usr/local/rtm/bin/rtm 49 > /dev/null 2> /dev/null)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19500]: (root) CMD (   sh /root/disk_stats.sh)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19510]: (root) CMD (/usr/local/directadmin/dataskq)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19508]: (root) CMD (/usr/local/directadmin/dataskq)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19502]: (root) CMD (/usr/local/rtm/bin/rtm 49 > /dev/null 2> /dev/null)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19503]: (root) CMD (   sh /root/disk_stats.sh)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19504]: (root) CMD (/usr/local/directadmin/dataskq)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19505]: (root) CMD (/usr/local/directadmin/dataskq)
Nov 13 16:55:42 s6 /USR/SBIN/CRON[19506]: (root) CMD (/usr/local/rtm/bin/rtm 49 > /dev/null 2> /dev/null)
(...)

As you can see, cron executes his jobs nonstop, many times in one second. I think it causes high load and my problems. Has someone any ideas how to solve this?

zEitEr · Nov 14, 2013

I guess, you should better get someone who investigate the issue and fix it for you. Please feel free to contact some of us here who posted in the thread. I'll be happy as well to assist you.

Lots of overload messages in one moment

Muniek

New member

nobaloney

NoBaloney Internet Svcs - In Memoriam †

Muniek

New member

Muniek

New member

nobaloney

NoBaloney Internet Svcs - In Memoriam †

Muniek

New member

nobaloney

NoBaloney Internet Svcs - In Memoriam †

Muniek

New member

zEitEr

Super Moderator

Muniek

New member

zEitEr

Super Moderator