apache high load problems

C

Chrysalis

Verified User
Joined
Aug 25, 2004
Messages
1,587
Location
uk
Hi recently I have been having problems with apache processes spawning in high amounts and saturating server load, it remains slow and unresponsive until httpd is restarted or a killall issued on the processes, sometimes it fixes itself when they all crash but this is usually after 2 hours or so.

In the error_log there are these entries.

httpd in free(): error: recursive call
httpd in free(): error: recursive call
httpd in free(): error: recursive call
httpd in free(): error: recursive call

over 50 altogether and about 30 of these

httpd in free(): error: page is already free
httpd in free(): error: page is already free
httpd in free(): error: page is already free

Following these there are tons of segfaults and abort traps when the child processes eventually died. When this happens the load avg goes above 50 and its happening almost every day now at least once a day, apache is 1.3.34 and php 4.4.0, it was 4.4.1 but I downgraded to see if it would resolve the issue.

I am worried there is some kind of easy way to ddos apache and the server is been attacked, or it could just be a bad script or something but the logs are giving me nothing and there is no increase in traffic and connections when it happens, os is FreeBSD 5.4.
 
for the benefit of others who may have same issue since this isnt answered on google, I have somewhat started tracking the problem and it seems to be down to mysql locking up and causing httpd to leave stale connections and build up of requests. Restarting apache helped because it removed all the pending requests from mysql allowing it to function again, I have changed my thread type in mysql and some settings in my.cnf to help resolve the issue, I will post more when I am satisfied the issue is resolved.
 
Hi,

Have you found the best settings for apache and mysql ?

I have high load server with httpd and mysqld ... my cpu is loaded at 800% ... it cause a timeout with directadmin...

I'm trying differents settings with my.cnf and apache but it's not easy...

for info :

AMD Sempron 2600+
512 MB RAM
2*80Go HDD
Fedora C3
MySQL 4.10a
PHP4

I'm hosting ~ 700 websites php/sql/html/perl
i haven't any application... only web hosting.

my settings :

my.cnf :

key_buffer = 256M
max_allowed_packet = 1M
table_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache = 8
query_cache_size = 16M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8


my apache config :

Timeout 300
KeepAlive On
MaxKeepAliveRequests 500
KeepAliveTimeout 5
MinSpareServers 5
MaxSpareServers 20
StartServers 6
MaxClients 450
MaxRequestsPerChild 1000
Click to expand...

any idea to help me please ? :(
 
8.00, or 800.0 ?

Either way, it's not 800%; it's the number of processes waiting to run during the last minute, the last five minutes, and the last 15 minutes.

The few times we've had similar problems it was always http or php (or both), sometimes because of an attack; sometimes run amok.

Either way, rebooting helped instantly.

While it's less messy (in terms of possibility of disk problems) to do a clean:

# shutdown -r now

from the prompt, it's not always possible. If you're at your server you can do a Ctrl-Alt-Delete from an attached keyboard, which will send an interrupt instantly; linux (perhaps unix as well) will catch the interrupt and do a clean shutdown, and you don't even have to be logged in.

Worst case is a power-interrupt reboot.

It hardly ever happens, and it's so infrequent that we don't believe we can eliminate it with some configuration changes.

Jeff
 
Well I think it was a combination of 2 things.

(a) mysql on fbsd isnt great it has problems with some of the threading routines that freebsd uses and so that was one thing I spent time on. I ended up disabling process scope threading and I think it is now using system scope (hard to tell since I havent found a way for mysql to report what threading it is using), but fbsd basically has a choice of either linuxthreads,libthr, and libpthread. In 4.x linuxthreads is known to be best and in 5.x and 6.x I believe they reccomend libpthread, however I have read reports that libpthread can lock up under heavy load and libthr is more stable. I think my mysql is using libpthread and the setting I changed seemed to have stopped the lockups, I have no idea how to make mysql use libthr in freebsd 5.x. If it didnt work I was going to switch to linuxthreads.

(b) hd thrashing, the server wasnt using tons of swap but it was using around 200meg, which probably caused some hd thrashing on mysql, some settings I changed on mysql which I believed helped was reducing cache sizes so it used less ram and wasnt caching to a swap file but actual ram instead.

Here is my current my.cnf but bear in mind the server has now had a ram upgrade so I reincreased some of the settings now it is all in physical ram.

[mysqld]
skip-locking
skip-innodb
query_cache_limit=8M
query_cache_size=256M
query_cache_type=1
max_connections=200
interactive_timeout=20
wait_timeout=20
connect_timeout=6
thread_cache_size=128
key_buffer=16M
join_buffer=1M
max_allowed_packet=16M
table_cache=1024
record_buffer=1M
sort_buffer_size=2M
read_buffer_size=2M
max_connect_errors=10

#log_warnings=9
net_retry_count=10

# Try number of CPU's*2 for thread_concurrency
thread_concurrency=4
myisam_sort_buffer_size=64M
#log-bin
server-id=1

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
open_files_limit=4096

[mysqldump]
quick
max_allowed_packet=16M

[mysql]
no-auto-rehash
#safe-updates

[isamchk]
key_buffer=32M
sort_buffer=32M
read_buffer=16M
write_buffer=16M

[myisamchk]
key_buffer=32M
sort_buffer=32M
read_buffer=16M
write_buffer=16M

#[mysqlhotcopy]
#interactive-timeout

I think many people might be tempted to raise things like key_buffer and assume it will pump up mysql's speed but if it saturates the server's ram I believe it will just have a negative affect and compromise stability.
 
Hi,

Thanks for yours responses :)

See the CPU overload at .http://www.atomysk.com/serverstatus.jpg

In the 'top', I can see httpds and mysqls servers who take 20, 40, 50 % of cpu each. So the cpu is very, very overloaded...
My websites are working fine (a tad slow) but the directadmin is totally down, see the result at : http://www.atomysk.com/da_timeout.jpg
However I've increased the timeout value in DA settings.....
Also, the SSH is unreachable... strange, nop ?

So...I'm now trying this settings :

skip-locking
key_buffer = 16M
table_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
max_connections = 64
max_connect_errors = 10
max_allowed_packet = 6M
myisam_sort_buffer_size = 64M
interactive_timeout = 20
wait_timeout = 20
connect_timeout = 6
record_buffer = 1M
read_rnd_buffer_size = 2M
thread_cache = 6
thread_cache_size = 32
query_cache_type = 1
query_cache_limit = 1M
query_cache_size = 32M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 2

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 32M
sort_buffer_size = 32M
read_buffer = 8M
write_buffer = 8M

[myisamchk]
key_buffer = 32M
sort_buffer_size = 32M
read_buffer = 6M
write_buffer = 6M

#[mysqlhotcopy]
#interactive-timeout
Click to expand...

and

Timeout 300
KeepAlive On
MaxKeepAliveRequests 400
KeepAliveTimeout 5
MinSpareServers 5
MaxSpareServers 10
StartServers 10
MaxClients 450
MaxRequestsPerChild 800
Click to expand...

I will see tonight... I hope...

Thanks again for help. Very thanks :)

Alba


EDIT : I notify I have the KISS firewall (with 2222 trusted of course) and the port of SSH and webmin have been changed and the DA mod_evasive is enable.
 
Last edited:
Regarding your apache settings I suggest disabling keepalive (more stable at cost of performance) reducing timeout to something like 60 instead of 300 and maxkeepalive to 150 although it should be irrelevant if keepalive is disabled.
 
Hi,

Thanks !

See :) http://193.24.215.99/mrtg/cpu-day.png

I've found some errors with glibc / gcc
A lot of errors in DA / exim / system logs...

I've first updated glibc and gcc, and now it working better. I continue to try to resolve the other remaining errors... tell me good luck ;)

good night,

Alba
 
/etc/my.cnf

and
/usr/local/directadmin/mysql.conf (i think)
 
alba said:
/etc/my.cnf

and
/usr/local/directadmin/mysql.conf (i think)
Click to expand...

Sorry, I also cannot locate. It's no file in /etc/my.cnf or usr/local/directadmin/mysql.conf

Thanks , any ideas?
 
you must create it in /etc/
create a file named my.cnf and write in to configure it.
 
I never hear my.cnf can run in the server before. Is it really will run it after I create the file?

my.cnf is the directadmin function or not? Sorry for stupid questions, since I using linux for a time but never hear can create my.cnf will config the server.
 
waterkei said:
I never hear my.cnf can run in the server before. Is it really will run it after I create the file?

my.cnf is the directadmin function or not? Sorry for stupid questions, since I using linux for a time but never hear can create my.cnf will config the server.
Click to expand...

Without an /etc/my.cnf file, MySQL will run with all default settings. It is not required, but if you want any tuning or non-default behaviour, then you need to create this file.
 
hi,

i've the same problem, freebsd 5.5-PRERELASE

how i have to conf my.cnf?

i've a Celeron 2.4Ghz with 1GB of Ram.

Can u paste me the correct configuration of my.cnf and if necessary the other configuration?

Thanks a lot
 
There's a copy of a good default my.cnf file posted on these forums; just do a search :) .

And don't forget to restart the mysqld daemon after you install it.

Jeff
 
Observing this problem as well...

Greetings everyone... I am observing what appears to be a similar problem as well...

I have a dedicated server with *very* light traffic. It looks like every day or so my server load slowly climbs from 0.20 to 14.00 and then stays there indefinitely.... locking down the system [until I do an "apachectl restart"] after which the server load drops back down.

I'm not getting any weird Fatal Errors from PHP or any Warnings [though a lot of PHP Notices.]. The apache httpd_error.log shows the following error, usually, when the slow climb to 14.00 server load begins:

httpd in free(): error: recursive call
httpd in free(): error: recursive call

[Usually it only shows about three or four of these repeated errors.]

I am using FreeBSD 6.2 and MySQL 4.1. I am trying to use the libthr threading mechanism through the libmap.conf setting, as Chrysalis alluded to earlier in this post, as a possible fix. [Though, I don't know if I have in fact been successful in switching to libthr or not... because I'm not sure if I need to recompile / reboot?]

In any event, my libmap.conf settings are now [located in /etc/libmap.conf]:

[mysqld]
libc_r.so libthr.so
libc_r.so.6 libthr.so.2
libthr.so.2 libthr.so.2
libpthread.so libthr.so
libpthread.so.2 libthr.so.2

and I also added ...
WITH_LIBMAP= yes

to my make.conf file.

Is there something else I need to do [e.g., recompile? / reboot?] in order to activate libthr?

Also, I'm not sure how much of what Chrysalis wrote above applies... for example, regarding "disabling process scope threading" or "hd thrashing" since it was two years ago... and now this is FreeBSD 6.2... but I would be greatly indebted to anyone who might be willing or able to help in this regards,

Blessings,
Albert Wong
www.ithou.org

PS. The mysql_logfile is completely empty... I don't know if that is unusual or not.

PPS. Here's the my.cnf settings for this machine:

[mysqld]
safe-show-database
skip-innodb
max_connections = 500
key_buffer = 32M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1800
thread_cache_size = 384
wait_timeout = 90
connect_timeout = 10
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 16M
max_connect_errors = 10
read_rnd_buffer_size = 524288
bulk_insert_buffer_size = 8M
query_cache_limit = 3M
query_cache_size = 80M
query_cache_type = 1
query_prealloc_size = 163840
query_alloc_block_size = 32768
skip-name-resolve

[mysqld_safe]
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 16M
sort_buffer = 16M
read_buffer = 16M
write_buffer = 16M

[mysqlhotcopy]
interactive-timeout

log = /var/log/mysql/mysql_logfile
 
hi to help reduce load averages on average webservers I reccomend the following.

disabling keepalive in apache
ensure max processes in apache isnt going to cause disk swapping
on freebsd 6.x usiing libthr instead of libpthread and dont use process scope on compile.
on freebsd 5.x use libpthread and compile with process scope
 
You must log in or register to reply here.
Back
Top