ServerLimit?

R

Rebecca

Verified User
Joined
May 20, 2006
Messages
36
Hello,

I've got a server hosting a very high traffic website/image hosting service. I seem to have hit some limit on processes that I can't quite sort out.

MRTG is showing a flat plateau on the Processes at 1107. I thought I was hitting a MaxClients limit since my open connections were right around the 1024 limit. So I set keepalive OFF (It had been previously been set to ON with a keepalive timeout of 1 second) and the open connections dropped down to around 800. But eventhough the connections dropped well below the 1024 limit I was still seeing a flat plateau at 1107 processes and the site was peforming poorly. There's no server-side stuff going on... just static html and serving of images.

Anyhow, I'm stumped. I've been reading about another parameter called ServerLimit and I thought that might have something to do with it, but I'm confused about what it does and how it's set. My httpd.conf doesn't even have anything listed for Server Limit:

Timeout 300
KeepAlive Off
MaxKeepAliveRequests 500
KeepAliveTimeout 1
MinSpareServers 5
MaxSpareServers 20
StartServers 8
MaxClients 1024
MaxRequestsPerChild 1000


Apache Error log doesn't show anything meaningful... just a few file not found errors.

Grateful for any suggestions...
Rebecca
 
Which *nix are you using and what's the kernel revision. There are other limits that are not within Apache itself from what I know, not that I know lot :D
 
Hmm... I don't know what *nix means...

But I'm running Apache 1.3.37 and Linux Kernel 2.6.9-42.0.10.EL

Rebecca
 
A thought.... maybe by *nix you mean what Operating System? (I don't speak geek too well!) It's CentOS.

Thanks!
Rebecca
 
Sorry about that. *nix was shortform to mean which Unix/Linux platform you are running on.

There used to be a 1024 threads per process limit in 2.4 but since you're using 2.6 (Centos 4?), it shouldn't be the case anymore.

But just to be safe, check by running cat /proc/sys/kernel/threads-max

Another thing is, how much memory does your server have?
 
Last edited:
Thanks,

I entered the code you suggested and got this number:
32734

I assume that means the maching can handle 32734 threads? I admit I'm a bit confused about the difference between threads and processes and tasks.

I'm wondering if I may be seeing a bandwidth limitation. When I look at my MRTG charts it seems that as soon as the bandwidth hits about 46-megabits per second, the number of TPC/IP connections suddenly jumps form about 700to around 1000. At the same time, the total number of process hits a very hard and absolute limit of about 1105.

Do you think that hitting a bandwidth limit could cause extra processes to suddenly be spawned?

Thanks,
Rebecca
 
A process is a running program/task. It can spawn child processes. Multi-threaded process can use threads to do work. At least that's how I understand it :D

It used to be that each process can only have up to 1024 threads.

Another limit is your server memory, each apache child process takes up memory. If say the average child takes 6MB (a quick look at my new unloaded server says 5.7 on the average), 1024 child processes would have used up 6GB of RAM.

You can check this by using ps -ylC httpd --sort:rss
 
Hmm... Well, I've got 1 Gig of memory. I seem to be getting some conflicting info in that department.

MRTG says that I'm using about 650 MB of memory

Top shows the following:

top - 23:31:58 up 34 days, 19:02, 1 user, load average: 0.42, 0.52, 0.42
Tasks: 1098 total, 1 running, 1097 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.8% us, 9.0% sy, 0.0% ni, 80.0% id, 4.2% wa, 0.0% hi, 0.0% si
Mem: 1034668k total, 1024084k used, 10584k free, 20704k buffers
Swap: 1052248k total, 17060k used, 1035188k free, 421752k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24879 root 17 0 3488 1556 760 R 2.2 0.2 0:01.35 top
12454 named 16 0 39212 2520 1340 S 0.3 0.2 108:05.40 named
21767 apache 15 0 22844 5424 1756 S 0.3 0.5 0:00.68 httpd
21972 apache 15 0 22776 5444 1768 S 0.3 0.5 0:00.69 httpd
22023 apache 16 0 22764 5424 1752 S 0.3 0.5 0:00.72 httpd
22293 apache 16 0 22776 5416 1748 S 0.3 0.5 0:00.69 httpd
22504 apache 16 0 22792 5428 1760 S 0.3 0.5 0:00.61 httpd
22509 apache 16 0 22764 5424 1748 S 0.3 0.5 0:00.58 httpd
22666 apache 15 0 22752 5388 1720 S 0.3 0.5 0:00.52 httpd
22795 apache 15 0 22768 5404 1736 S 0.3 0.5 0:00.43 httpd
22806 apache 15 0 22772 5416 1740 S 0.3 0.5 0:00.57 httpd
22825 apache 16 0 22808 5404 1736 S 0.3 0.5 0:00.59 httpd
23136 apache 15 0 22768 5444 1772 S 0.3 0.5 0:00.55 httpd
with many more lines of this...

It looks to me like Top thinks I'm using more memory than MRTG does.

I tried typing in what you suggested and got the following... but I don't know what it means.



S 100 22122 10671 0 76 0 5472 5709 - ? 00:00:00 httpd
S 100 22147 10671 0 75 0 5472 5734 - ? 00:00:00 httpd
S 100 22327 10671 0 76 0 5472 5707 - ? 00:00:00 httpd
S 100 22344 10671 0 76 0 5472 5704 - ? 00:00:00 httpd
S 100 22615 10671 0 76 0 5472 5704 394522 ? 00:00:00 httpd
S 100 22973 10671 0 75 0 5472 5700 - ? 00:00:00 httpd
S 100 23263 10671 0 76 0 5472 5708 - ? 00:00:00 httpd
S 100 23597 10671 0 76 0 5472 5710 - ? 00:00:00 httpd
S 100 23643 10671 0 75 0 5472 5716 - ? 00:00:00 httpd
S 100 24187 10671 0 76 0 5472 5707 - ? 00:00:00 httpd
S 100 21594 10671 0 76 0 5476 5708 - ? 00:00:00 httpd
S 100 22310 10671 0 75 0 5476 5709 - ? 00:00:00 httpd
S 100 22465 10671 0 76 0 5476 5703 - ? 00:00:00 httpd
S 100 22502 10671 0 76 0 5476 5707 - ? 00:00:00 httpd
S 100 22632 10671 0 76 0 5476 5710 - ? 00:00:00 httpd
S 100 22643 10671 0 76 0 5476 5714 - ? 00:00:00 httpd
S 100 22657 10671 0 76 0 5476 5713 - ? 00:00:00 httpd
S 100 22786 10671 0 75 0 5476 5706 - ? 00:00:00 httpd
S 100 23578 10671 0 76 0 5476 5729 - ? 00:00:00 httpd
S 100 24575 10671 0 76 0 5476 5724 - ? 00:00:00 httpd
S 100 21719 10671 0 76 0 5480 5709 - ? 00:00:00 httpd
S 100 22164 10671 0 76 0 5480 5708 - ? 00:00:00 httpd
S 100 22950 10671 0 76 0 5480 5710 select ? 00:00:00 httpd
S 100 23443 10671 0 75 0 5480 5717 - ? 00:00:00 httpd
S 100 23640 10671 0 76 0 5480 5715 - ? 00:00:00 httpd
S 100 21921 10671 0 76 0 5484 5710 - ? 00:00:00 httpd
S 100 21982 10671 0 76 0 5484 5709 - ? 00:00:00 httpd
S 100 21983 10671 0 76 0 5484 5734 - ? 00:00:00 httpd
S 100 22302 10671 0 76 0 5484 5709 - ? 00:00:00 httpd
S 100 22479 10671 0 76 0 5484 5712 - ? 00:00:00 httpd
S 100 22790 10671 0 76 0 5484 5717 - ? 00:00:00 httpd
S 100 23455 10671 0 76 0 5484 5707 - ? 00:00:00 httpd
S 100 23617 10671 0 76 0 5484 5720 - ? 00:00:00 httpd
S 100 21157 10671 0 76 0 5488 5725 - ? 00:00:00 httpd
S 100 21591 10671 0 75 0 5488 5734 - ? 00:00:00 httpd
S 100 22308 10671 0 76 0 5488 5704 - ? 00:00:00 httpd
S 100 24398 10671 0 75 0 5488 5720 select ? 00:00:00 httpd
S 100 24503 10671 0 76 0 5488 5717 - ? 00:00:00 httpd
S 100 21945 10671 0 75 0 5492 5725 - ? 00:00:00 httpd
S 100 22134 10671 0 76 0 5492 5707 - ? 00:00:00 httpd
S 100 22136 10671 0 76 0 5492 5715 - ? 00:00:00 httpd
S 100 22179 10671 0 75 0 5492 5725 - ? 00:00:00 httpd
S 100 22311 10671 0 76 0 5492 5709 - ? 00:00:00 httpd
S 100 22315 10671 0 76 0 5492 5710 - ? 00:00:00 httpd
S 100 22639 10671 0 76 0 5492 5709 - ? 00:00:00 httpd
S 100 21966 10671 0 76 0 5496 5734 - ? 00:00:00 httpd
S 100 23120 10671 0 75 0 5496 5734 - ? 00:00:00 httpd
S 100 24114 10671 0 76 0 5496 5734 - ? 00:00:00 httpd
S 100 21204 10671 0 76 0 5508 5734 - ? 00:00:00 httpd
S 100 21760 10671 0 75 0 5512 5733 - ? 00:00:00 httpd
S 100 21938 10671 0 75 0 5524 5717 - ? 00:00:00 httpd
S 100 22126 10671 0 76 0 5524 5734 - ? 00:00:00 httpd
S 100 22291 10671 0 75 0 5528 5731 - ? 00:00:00 httpd
S 100 23300 10671 0 76 0 5548 5733 - ? 00:00:00 httpd
S 100 22649 10671 0 76 0 5552 5734 - ? 00:00:00 httpd
S 100 23109 10671 0 75 0 5568 5734 select ? 00:00:00 httpd
S 100 21596 10671 0 76 0 5576 5734 - ? 00:00:00 httpd
S 100 22140 10671 0 76 0 5576 5729 - ? 00:00:00 httpd
S 0 10671 1 0 76 0 13304 5687 select ? 00:00:13 httpd


Thanks again for your help!
Rebecca
 
I might be barking up the wrong tree since none of my sites or client's are that popular. But it doesn't seem like you're that out of memory yet, since swap is largely unused. My guess is there's some system wide limit on 1024 connections or something that neither of us are aware of, and none of the experts are voicing up either :D

But if you feel like giving it a shot, you might want to consider upgrading to apache 2 because it has something called MPM that supposedly is much better for sites that see a lot of connections.

In apache 1.3 and 2 that uses prefork, each thread handles 1 connection but each browser can hit your site with several connection.

With MPM, each child thread handles several connections resulting in less memory use.
 
You must log in or register to reply here.
Back
Top