Please need help reducing server load on a new dedicated server?

beansbaxter · May 15, 2013

I really need some help please.

I purchase a new Dell R420 dedicated server and am trying to get optimized better. Server load is always in the 4's and 5's and peaks into the 8's and 10's at times. The CPU usage is being consumed by MySQL and Apache. I have ample processing power and memory, so why is server load so high?

DA shows 24 processors, but here is the physical hardware tech specs:

Processors - Dual Intel Xeon E5-2420 1.90GHz, 15M Cache, 7.2GT/s QPI, Turbo, 6C, 95W
Memory - 32GB Total, 4 x 8GB RDIMM, 1333 MT/s, Low Volt, Dual Rank, x4 Data Width
Hard drives - 300GB 10K RPM SAS 6Gbps 2.5in

On the software side, I'm running CentOS 5.9 with DA 1.43 and the following services:

Apache 2.2.24
MySQL 5.1.65
Php 5.2.17

The server is only running 6 websites, with each website having its own MySQL database. Of the six databases being used for the 6 sites:

one database is very large (over 4GB)
one database is medium (under 2GB)
the rest are small databases (under 1GB)

Output from "top c" and then sorted by processes that are currently consuming the most CPU.

Code:

top - 12:23:07 up 8 days,  3:15,  1 user,  load average: 5.17, 4.88, 4.78
Tasks: 427 total,   4 running, 423 sleeping,   0 stopped,   0 zombie
Cpu(s): 18.8%us,  0.7%sy,  0.0%ni, 80.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32915676k total,  7178832k used, 25736844k free,   204620k buffers
Swap: 67111528k total,        0k used, 67111528k free,  5967424k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                
23498 mysql     15   0  373m  63m 4980 S 148.5  0.2   9264:39 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql -
28638 apache    15   0  173m  30m 4400 R 53.4  0.1   0:12.50 /usr/sbin/httpd -k start -DSSL                          
  338 apache    16   0  110m  19m 3796 R 51.4  0.1   0:04.11 /usr/sbin/httpd -k start -DSSL                          
30767 apache    16   0  173m  30m 4216 S 31.5  0.1   0:18.40 /usr/sbin/httpd -k start -DSSL                          
31757 apache    16   0  111m  22m 4012 S 28.5  0.1   0:10.61 /usr/sbin/httpd -k start -DSSL                          
28163 apache    15   0  174m  30m 4396 S 24.9  0.1   0:31.68 /usr/sbin/httpd -k start -DSSL                          
30772 apache    15   0  167m  23m 4248 S 23.9  0.1   0:05.31 /usr/sbin/httpd -k start -DSSL                          
29957 apache    16   0  120m  30m 3936 R 22.2  0.1   0:30.02 /usr/sbin/httpd -k start -DSSL                          
  374 apache    15   0  162m  20m 3836 S 14.9  0.1   0:02.11 /usr/sbin/httpd -k start -DSSL                          
32323 apache    16   0  165m  22m 4224 S 14.6  0.1   0:03.27 /usr/sbin/httpd -k start -DSSL                          
 1371 apache    15   0  106m  18m 3464 S 11.6  0.1   0:00.87 /usr/sbin/httpd -k start -DSSL                          
 1314 apache    16   0  162m  20m 3780 S  8.0  0.1   0:01.47 /usr/sbin/httpd -k start -DSSL                          
 1372 apache    15   0  108m  19m 3392 S  8.0  0.1   0:00.30 /usr/sbin/httpd -k start -DSSL                          
31355 apache    15   0  113m  23m 4028 S  6.0  0.1   0:15.26 /usr/sbin/httpd -k start -DSSL                          
30726 apache    15   0  165m  22m 4500 S  4.6  0.1   0:05.91 /usr/sbin/httpd -k start -DSSL                          
 1373 apache    15   0  107m  17m 3928 S  4.0  0.1   0:00.38 /usr/sbin/httpd -k start -DSSL                          
 1630 apache    18   0  100m  12m 3312 S  4.0  0.0   0:00.12 /usr/sbin/httpd -k start -DSSL                          
21224 apache    15   0  165m  22m 4320 S  2.0  0.1   0:24.43 /usr/sbin/httpd -k start -DSSL                          
30729 apache    15   0  113m  23m 4076 S  2.0  0.1   0:05.03 /usr/sbin/httpd -k start -DSSL                          
30770 apache    15   0  166m  23m 4236 S  2.0  0.1   0:05.97 /usr/sbin/httpd -k start -DSSL                          
 1137 root      15   0 13024 1404  836 R  0.7  0.0   0:05.84 top c                                                   
30466 apache    15   0  173m  30m 4284 S  0.7  0.1   0:07.97 /usr/sbin/httpd -k start -DSSL                          
   20 root      RT  -5     0    0    0 S  0.3  0.0   1:38.45 [migration/6]                                           
14712 root      18   0 98936 8244 3536 S  0.3  0.0   0:03.34 /usr/sbin/httpd -k start -DSSL                          
    1 root      15   0 10368  640  544 S  0.0  0.0   0:12.99 init [3]                                                
    2 root      RT  -5     0    0    0 S  0.0  0.0  20:18.56 [migration/0]                                           
    3 root      34  19     0    0    0 S  0.0  0.0   0:01.77 [ksoftirqd/0]                                           
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/0]                                            
    5 root      RT  -5     0    0    0 S  0.0  0.0   2:39.52 [migration/1]                                           
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/1]                                           
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/1]                                            
    8 root      RT  -5     0    0    0 S  0.0  0.0   2:37.04 [migration/2]                                           
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/2]                                           
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/2]                                            
   11 root      RT  -5     0    0    0 S  0.0  0.0   1:58.87 [migration/3]                                           
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.02 [ksoftirqd/3]                                           
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/3]                                            
   14 root      RT  -5     0    0    0 S  0.0  0.0   2:05.00 [migration/4]                                           
   15 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/4]                                           
   16 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 [watchdog/4]                                            
   17 root      RT  -5     0    0    0 S  0.0  0.0   1:41.43 [migration/5]                                           
   18 root      34  19     0    0    0 S  0.0  0.0   0:00.01 [ksoftirqd/5]

Output from "free -m"

Code:

             total       used       free     shared    buffers     cached
Mem:         32144       6986      25157          0        199       5818
-/+ buffers/cache:        968      31175
Swap:        65538          0      65538

Output from "iostat -x 1 10"

Code:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.86    0.00    0.54    0.02    0.00   82.58

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.04    65.99  0.89 26.90    15.89   743.18    27.32     0.05    1.62   0.24   0.67
sda1              0.00     0.00  0.00  0.00     0.00     0.00    25.27     0.00    3.92   2.81   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00    33.43     0.00    3.98   2.64   0.00
sda3              0.04    65.99  0.89 26.90    15.89   743.18    27.32     0.05    1.62   0.24   0.67
sdb               0.00     0.00  0.00  0.00     0.00     0.00    19.10     0.00    4.00   3.79   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00    17.26     0.00    3.48   3.10   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.12    0.00    0.62    0.00    0.00   83.26

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.20    0.00    0.62    0.00    0.00   87.18

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.89    0.00    0.79    0.00    0.00   81.32

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   190.00  0.00 68.00     0.00  2064.00    30.35     0.03    0.41   0.01   0.10
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00   190.00  0.00 68.00     0.00  2064.00    30.35     0.03    0.41   0.01   0.10
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.04    0.00    0.46    0.00    0.00   87.51

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.69    0.00    0.58    0.00    0.00   87.72

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  5.00     0.00    40.00     8.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  5.00     0.00    40.00     8.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          13.07    0.00    0.67    0.00    0.00   86.26

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.71    0.00    0.88    0.12    0.00   81.29

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  4.00  0.00    32.00     0.00     8.00     0.03    8.25   8.25   3.30
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00     0.00  4.00  0.00    32.00     0.00     8.00     0.03    8.25   8.25   3.30
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          21.92    0.00    0.92    0.04    0.00   77.12

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   254.00  0.00 531.00     0.00  6280.00    11.83     0.93    1.76   0.02   0.90
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00   254.00  0.00 531.00     0.00  6280.00    11.83     0.93    1.76   0.02   0.90
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          29.28    0.00    1.17    0.00    0.00   69.55

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

explosive · May 15, 2013

First update software, you have some quite old stuff: install PHP 5.4, Apache 2.4.4, MySQL 5.6 - all 3 have major performance improvements. Next: php via php-fpm (or for while by mod_fcgid because needed patches for fpm setup in in beta).

chatwizrd · May 15, 2013

Do you use any raid on your hard drives?

Have you tried enable logging slow mysql queries?

Have you edited the mysql config /etc/my.cnf at all?

Have you tried mytop before to see what queries are taking so long?

How many active connections does the site have? Enable extended status in apache.

beansbaxter · May 15, 2013

chatwizrd said:
Do you use any raid on your hard drives?.

The hard drives are powered by the following RAID controller, a PERC H710 Integrated RAID Controller with 512MB NV Cache.

I am running a RAID 1 mirror array with an additional hot spare.

chatwizrd said:
Have you tried enable logging slow mysql queries?

I haven't tried this. I need to figure out how to do this and see what it says.

chatwizrd said:
Have you edited the mysql config /etc/my.cnf at all?

I have only added one line to this file. Here is the contents of /etc/my.cnf:

Code:

[mysqld]
local-infile=0

max_allowed_packet=128M

chatwizrd said:
Have you tried mytop before to see what queries are taking so long?

I tried to install it, but I'm getting errors when trying to run it. Output below:

Code:

[root@server ~]# cd mytop
[root@server mytop]# ls
mytop-1.6  mytop-1.6.tar.gz
[root@server mytop]# cd mytop-1.6
[root@server mytop-1.6]# ls
Changes  INSTALL  Makefile.PL  MANIFEST  META.yml  mytop  README  test.pl
[root@server mytop-1.6]# perl Makefile.PL
Checking if your kit is complete...
Looks good
Warning: prerequisite DBD::mysql 1 not found.
Warning: prerequisite Term::ReadKey 2.1 not found.
Writing Makefile for mytop
[root@server mytop-1.6]# ls
Changes  Makefile     MANIFEST	mytop	test.pl
INSTALL  Makefile.PL  META.yml	README
[root@server mytop-1.6]# make
cp mytop blib/script/mytop
/usr/bin/perl "-MExtUtils::MY" -e "MY->fixin(shift)" blib/script/mytop
Manifying blib/man1/mytop.1
[root@server mytop-1.6]# make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-Iblib/lib" "-Iblib/arch" test.pl
1..1
ok 1
[root@server mytop-1.6]# make install
Installing /usr/share/man/man1/mytop.1
Installing /usr/bin/mytop
Writing /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/mytop/.packlist
Appending installation info to /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/perllocal.pod
[root@server mytop-1.6]# mytop
Can't locate Term/ReadKey.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at /usr/bin/mytop line 175.

chatwizrd said:
How many active connections does the site have? Enable extended status in apache.

How do I determine the number of active connections?

Regarding extended status in Apache, let me figure out how to do this and report back.

Thanks.

beansbaxter · May 15, 2013

explosive said:
First update software, you have some quite old stuff: install PHP 5.4, Apache 2.4.4, MySQL 5.6 - all 3 have major performance improvements. Next: php via php-fpm (or for while by mod_fcgid because needed patches for fpm setup in in beta).

Yeah, I know there are newer versions available, however I need to make sure the software powering the websites can/will run on these newer versions. I'm less concerned about MySQL and Apache, and more concerned about PHP because last time I checked, the sites didn't work in 5.3

I would think that this kind of processing power should have no problem with these older versions of php though.

Looking into the newest versions of Apache and MySQL now.

Thanks.

Please need help reducing server load on a new dedicated server?

beansbaxter

Verified User

explosive

Verified User

chatwizrd

Verified User

beansbaxter

Verified User

beansbaxter

Verified User