Help identify CPU and memory usage spikes

S

simba

Verified User
Joined
Oct 13, 2012
Messages
54
Hello everyone. I get random RAM and CPU usage spikes, reaching memory limit and causing problems.

I cannot figure what is causing it, so i am asking your help. It is usually happening in the middle of the night. Logs look normal.

Today's server load:

day_load.png


Week's server load:

week_load.png


Today's CPU usage:

day_cpu.png


Week's CPU usage:

week_cpu.png


Today's RAM usage:

day_ram.png


Week's RAM usage:

week_ram.png


I have CSF which one time alerted me that one user is using too much memory sending following snapshot:

PS output:
Code: 
Output from ps:
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
 root         1  0.0  0.0  19232  1464 ?        Ss   Dec17   0:02 init
 root      1143  0.0  0.0  10664   576 ?        S<s  Dec17   0:00 /sbin/udevd -d
 root      1407  0.0  0.0 249280  3036 ?        Sl   Dec17   0:01 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
 named     1420  0.0  0.4 162816 15332 ?        Ssl  Dec17   0:00 /usr/sbin/named -u named
 root      1438  0.0  0.0  22524  1260 ?        Ss   Dec17   0:02 /usr/sbin/dovecot
 dovecot   1442  0.0  0.0  44644  2968 ?        S    Dec17   0:00  \_ dovecot/pop3-login
 dovecot   1444  0.0  0.0  44644  2964 ?        S    Dec17   0:00  \_ dovecot/pop3-login
 dovecot   1445  0.0  0.0  44644  2964 ?        S    Dec17   0:00  \_ dovecot/pop3-login
 dovecot   1464  0.0  0.0  44648  2968 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1465  0.0  0.0  44648  2972 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1469  0.0  0.0  44648  2972 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1470  0.0  0.0  44648  2972 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1471  0.0  0.0  44648  2968 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1472  0.0  0.0  44648  2968 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1473  0.0  0.0  44648  2972 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1474  0.0  0.0  44648  2968 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1475  0.0  0.0  44648  2968 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1477  0.0  0.0  44648  2968 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1478  0.0  0.0  44648  2968 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1479  0.0  0.0  44648  2972 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1480  0.0  0.0  44648  2972 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1481  0.0  0.0  44648  2972 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1482  0.0  0.0  44648  2972 ?        S    Dec17   0:00  \_ dovecot/imap-login
 dovecot   1484  0.0  0.0  16100  1184 ?        S    Dec17   0:00  \_ dovecot/anvil [33 connections]
 root      1485  0.0  0.0  16232  1264 ?        S    Dec17   0:00  \_ dovecot/log
 root      1487  0.0  0.0  19184  2036 ?        S    Dec17   0:02  \_ dovecot/config
 root      1491  0.0  0.0  21568  2276 ?        S    Dec17   0:03  \_ dovecot/auth [0 wait, 0 passdb, 0 userdb]
 dovecot  27965  0.0  0.0  44648  2964 ?        S    Dec21   0:00  \_ dovecot/imap-login
 dovecot   7772  0.0  0.0  44644  2964 ?        S    Dec26   0:00  \_ dovecot/pop3-login
 dovecot   7895  0.0  0.0  44644  2964 ?        S    Dec26   0:00  \_ dovecot/pop3-login
 dovecot  19584  0.0  0.0  44644  2964 ?        S    Dec26   0:00  \_ dovecot/pop3-login
 dovecot  11283  0.0  0.0  44644  2960 ?        S    Dec26   0:00  \_ dovecot/pop3-login
 dovecot  11391  0.0  0.0  44644  2964 ?        S    Dec26   0:00  \_ dovecot/pop3-login
 dovecot  11583  0.0  0.0  44644  2960 ?        S    Dec26   0:00  \_ dovecot/pop3-login
 dovecot  11685  0.0  0.0  44644  2960 ?        S    Dec26   0:00  \_ dovecot/pop3-login
 dovecot  11905  0.0  0.0  44644  2960 ?        S    Dec26   0:00  \_ dovecot/pop3-login
 dovecot  13340  0.0  0.0  44644  2964 ?        S    Dec26   0:00  \_ dovecot/pop3-login
 dovecot  21621  0.0  0.0  44644  2964 ?        S    Dec26   0:00  \_ dovecot/pop3-login
 dovecot  11548  0.0  0.0  44644  2964 ?        S    Dec27   0:00  \_ dovecot/pop3-login
 dovecot  26092  0.0  0.0  44644  2964 ?        S    Dec27   0:00  \_ dovecot/pop3-login
 dovecot  26282  0.0  0.0  44644  2960 ?        S    Dec27   0:00  \_ dovecot/pop3-login
 root      1466  0.0  0.0  64080  1192 ?        Ss   Dec17   0:00 /usr/sbin/sshd
 root      1494  0.0  0.0  22096   960 ?        Ss   Dec17   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
 root      1504  0.0  0.0 108172  1624 ?        S    Dec17   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/e6.lt.pid
 mysql     1768  0.6  1.7 920384 55528 ?        Sl   Dec17  92:57  \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/e6.lt.err --pid-file=/var/lib/mysql/e6.lt.pid --socket=/var/lib/mysql/mysql.sock --port=3306
 root      1969  0.0  0.0  18848   856 ?        Ss   Dec17   0:00 /usr/local/directadmin/da-popb4smtp
 nobody    1976  0.0  0.0  54688  2604 ?        Ss   Dec17   0:00 /usr/local/directadmin/directadmin d
 nobody   29796  0.0  0.0  54688  1156 ?        S    Dec26   0:00  \_ /usr/local/directadmin/directadmin d
 nobody   29804  0.0  0.0  54688  1156 ?        S    Dec26   0:00  \_ /usr/local/directadmin/directadmin d
 nobody   29808  0.0  0.0  54688  1156 ?        S    Dec26   0:00  \_ /usr/local/directadmin/directadmin d
 nobody   15930  0.0  0.0  54688  1156 ?        S    01:05   0:00  \_ /usr/local/directadmin/directadmin d
 nobody   15931  0.0  0.0  54688  1156 ?        S    01:05   0:00  \_ /usr/local/directadmin/directadmin d
 mail      1983  0.0  0.0  61088  1416 ?        Ss   Dec17   0:00 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
 root      2019  0.0  0.0 140640  1876 ?        Ss   Dec17   0:00 pure-ftpd (SERVER)                                                                                                                                                                              
 root      2026  0.0  0.0 117248  1304 ?        Ss   Dec17   0:01 crond
 root      2038  0.0  0.2 803376  8864 ?        Sl   Dec17   0:08 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -x
 root     11873  0.0  0.8 176260 27048 ?        Ss   00:00   0:05 lfd - sleeping
 root     12045  0.0  0.8 176260 25928 ?        S    05:08   0:00  \_ lfd - (child) checking load...
 root     12046  0.0  0.0 110344  1160 ?        R    05:08   0:00      \_ /bin/ps axuf
 root     25839  0.0  0.3 144188 10636 ?        Ss   03:21   0:00 /usr/sbin/httpd -k start -DSSL
 apache   25843  0.0  0.5 244460 16200 ?        S    03:21   0:02  \_ /usr/sbin/httpd -k start -DSSL
 apache   25844  0.0  0.5 246540 15960 ?        S    03:21   0:03  \_ /usr/sbin/httpd -k start -DSSL
 apache   25845  0.0  0.5 248408 17772 ?        S    03:21   0:02  \_ /usr/sbin/httpd -k start -DSSL
 apache   25855  0.0  0.5 246856 16236 ?        S    03:21   0:03  \_ /usr/sbin/httpd -k start -DSSL
 apache    1704  0.3  0.4 147900 14440 ?        S    05:03   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    1769  0.4  0.4 149228 14976 ?        S    05:03   0:01  \_ /usr/sbin/httpd -k start -DSSL
 apache    1780  0.2  0.3 144404 10280 ?        S    05:03   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    3090  0.3  0.4 147900 13360 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    3093  0.1  0.3 144404 10248 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    3096  0.1  0.3 144404 10308 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5188  0.2  0.4 147900 13268 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5222  0.1  0.3 144404 10280 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5226  0.1  0.4 147900 13296 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5227  0.1  0.4 147900 13316 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5354  0.1  0.3 144404 10252 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5370  0.2  0.4 147900 13324 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5408  0.1  0.3 144404 10280 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5416  0.1  0.3 144404 10292 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5425  0.2  0.4 147900 13288 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5426  0.1  0.3 144404 10256 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5429  0.1  0.3 144404 10296 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5443  0.1  0.3 144404 10280 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    5454  0.1  0.3 144404 10252 ?        S    05:04   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    7483  0.1  0.3 144404 10280 ?        S    05:05   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    7890  0.1  0.3 144404 10380 ?        S    05:05   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    8025  0.1  0.3 144404 10280 ?        S    05:05   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9338  0.1  0.3 144404 10256 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9653  0.2  0.4 147900 13264 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9699  0.1  0.0      0     0 ?        Z    05:06   0:00  \_ [httpd] <defunct>
 apache    9700  0.4  0.4 147900 13252 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9717  0.1  0.3 144404 10252 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9718  0.3  0.4 147900 13256 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9719  0.1  0.3 144404 10248 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9721  0.3  0.4 147900 13232 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9723  0.1  0.3 144404 10256 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9753  0.2  0.3 144404 10252 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9754  0.1  0.3 144404 10224 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9756  0.1  0.3 144404 10252 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9758  0.1  0.3 144404 10256 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9759  0.2  0.3 144404 10224 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9761  0.1  0.3 144404 10252 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9763  0.2  0.3 144404 10224 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9772  0.1  0.3 144404 10240 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9773  0.2  0.4 147900 13228 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9774  0.1  0.3 144404 10224 ?        S    05:06   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache    9836  0.2  0.3 144404 10252 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   10058  0.2  0.3 144404 10332 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11442  0.1  0.3 144404 10224 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11450  0.1  0.3 144404 10224 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11451  0.1  0.3 144404 10220 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11474  0.1  0.3 144404 10224 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11475  0.1  0.3 144404 10224 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11476  0.1  0.3 144404 10224 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11498  0.0  0.3 144404 10220 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11499  0.1  0.3 144404 10220 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11500  0.3  0.3 144404 10252 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11501  0.1  0.3 144404 10272 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11502  0.0  0.3 144404 10220 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11503  0.1  0.3 144404 10280 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11504  0.1  0.3 144404 10224 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11693  0.0  0.3 144404 10220 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11708  0.0  0.3 144404 10220 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 josh     11709  0.5  0.4 149184 14296 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11731  0.0  0.3 144404 10220 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11732  0.1  0.3 144404 10224 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11733  0.2  0.3 144404 10224 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11734  0.1  0.3 144404 10224 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL
 apache   11962  0.0  0.3 144476 10132 ?        S    05:07   0:00  \_ /usr/sbin/httpd -k start -DSSL

VMstat ouput:

Code: 
Output from vmstat:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
  0  0      0 2210796      0      0    0    0     7    38    0    1  2  0 97  0  0

But it happened only once, and i get spikes more often now.

I would appreciate any help. Thank you.
 
Last edited:
Hello,

You should check ps/top/htop output immediately as you notice spikes, there is no chance to guess what the reason is in your particular case.

High User CPU might be caused by any running software on your server including but not limited to Apache, MySQL, Exim, Dovecot, ProFTPd. I'd guess it's Apache+PHP+MySQL related, as they are the most common reason.
 
Thanks for your response!

I was not able to find software which would make a snapshot when there is increase in CPU and memory usage, could anybody recommend it?
 
Something else to check: if it's happening at the same time every night, and if you're making nightly backups, do you have enough memory to handle the backup load? If not, then you could be using memory cache during backups, which could cause the problem.

Jeff
 
You must log in or register to reply here.
Back
Top