APC tuning & flaw / instability

[soulshepard]

Dear collegues,

i installed a high traffic webshop on a directadmin server
(average 300-500 php session at any given time)

i have sometimes a deadlock situation if the cache is cleared by the tally cron or an apache httpd restart

then with in a minute all these 500 sessions are requesting something that is not in the cache and all try to fill the cache and wait to the php process to complete...

the server is on high cpu utilisation and slowly starts swapping and comes into a deadlock it never recovers, until you reboot or power on/off

in the logfiles sometimes i see cache_slam events but i tought these would prevent high cpu instead of creating them... so it must be something else...
all i can see on the server at that moment is very high cpu and tons of apache processes (that are trying to execute php)

howto avoid this on directadmin.. to disable the tally? or preload the cache somehow when httpd restarts? or some other method of ramping it up to get get cache quickly filled again before too many clients fireup php processes and wait for them to complete.. which is in my case never

to truely prevent this from happening, is to not clear the cache on high utilisation, but high utilisation i have all the time ....
so i guess i need to disable the tally crons... ?

i assume i should get this working with apc?
any other persons here willing to share there settings of apc on a high traffic server with apc?
purhaps some of my settings of apc are not correct?
or do i miss soemthing totally else?

i am considering memcache as a valid replacement. but pure because the cache is then not cleared and it is distributed.. .. (until you restart all of your memcached servers ofcource..)

als one option would be fastcgi so that you will not have maby php processes loading all the time, and becasue it its in memory it will be faster compiling and faster cache filling thus avoiding the deadlock...

anyone can help me with these toughts or share its apc config tuning config?



Thanks in advance..

atatched is a screenshot of the apc.php also


hardware:
to get this performing i have a centos 64bit quadcore 2 ghz 8gb and the disks are a sata2 mirror

the database server is seperate on another dedicated server also centos 8gb with a data disk mirror

software:
i have installed apc 3.1.3p1 together with directadmin (thus php 5.2.12)

fragmentation:
Fragmentation: 25.28% ( 40.2 MBytes out of 158.9 MBytes in 446 fragments)
sometimes high sometimes low
but my cache hit is a longer time is 100%

apc settings:

Server Software Apache/2
Shared Memory 1 Segment(s) with 256.0 MBytes
(mmap memory, pthread mutex locking)
Start Time 2010/02/02 17:44:10
Uptime 1 hour and 6 minutes


Runtime Settings
apc.cache_by_default 1
apc.canonicalize 1
apc.coredump_unmap 0
apc.enable_cli 0
apc.enabled 1
apc.file_md5 0
apc.file_update_protection 2
apc.filters
apc.gc_ttl 3600
apc.include_once_override 0
apc.lazy_classes 0
apc.lazy_functions 0
apc.max_file_size 50M
apc.mmap_file_mask
apc.num_files_hint 1000
apc.preload_path
apc.report_autofilter 0
apc.rfc1867 0
apc.rfc1867_freq 0
apc.rfc1867_name APC_UPLOAD_PROGRESS
apc.rfc1867_prefix upload_
apc.rfc1867_ttl 3600
apc.shm_segments 2
apc.shm_size 256
apc.stat 1
apc.stat_ctime 0
apc.ttl 7200
apc.use_request_time 1
apc.user_entries_hint 4096
apc.user_ttl 7200
apc.write_lock 1

File Cache Information
Cached Files 1645 ( 76.5 MBytes)
Hits 1833031
Misses 3476
Request Rate (hits, misses) 471.38 cache requests/second
Hit Rate 470.49 cache requests/second
Miss Rate 0.89 cache requests/second
Insert Rate 0.42 cache requests/second
Cache full count 0

User Cache Information
Cached Variables 2183 ( 19.4 MBytes)
Hits 519927
Misses 9415
Request Rate (hits, misses) 135.87 cache requests/second
Hit Rate 133.45 cache requests/second
Miss Rate 2.42 cache requests/second
Insert Rate 4.67 cache requests/second
Cache full count 0

 

[soulshepard]

on the high fragmentation i found two hints: (not relevant to the problem but an explenation why mine could be so fragmented.)

relevant post with other software on fragmentation.

http://lists.ez.no/pipermail/components/2009-January/004007.html

I'm caching every page in the site, for 1 year, and a few minutes after
startup the framentation starts and increases gradually until the server
goes down. I don't understand why there is fragmentation to begin with,
the cache is not full, the cache lifetime is 1 year long so nothing
should get tossed out. The only information i found online describes
fragmentation when the cache is full.

Clearing the cache from the apc.php file or through the method in the
compotent clears the used memory, but the fragmented pieces (gray ones
in the graph) stay gray. Only reloading apache will reset everything.
.....

answer:
http://lists.ez.no/pipermail/components/2009-January/004010.html

I just spoke to the guy, and apparently the issue is a too high rate of
insertions by multiple processes concurrently. If every process uses the
same APC key to store, they conflict and only one of them wins and the
other leave a "hole" in the memory space. This should not take the
server down though.
....