I am pretty much lost with this one. I have been at it for almost a week with this and everything I try just doesn't seem to work.
I just got a new server with CentOS 4.4. It was running fine, no issues for a few weeks, and then I transferred a pretty large vBulletin website over to it (170 users are usually on at any given time) and that's when the issues started. What happens is the server will run fine, great load (usually in between .5 and 1.2) then at random the load peaks and basically everything comes to a halt within minutes. When I say peaks, I mean I have actually seen it at 200+. At first my httpd error_log would be filled with "[error] child process 32530 still did not exit, sending a SIGKILL". I was getting so tired of having to submit a reboot ticket to my provider to have them do a hard reboot so I made a little script that checks the load every minute, if it's above 10 it restarts httpd. Then takes about a minute for the load to come back down assuming the script ran in enough time for it to enter the command. Now that the script runs that message in error_log isn't there, just says that httpd was restarted. No other logs looks suspicous at all and don't help at all.
Now I can't figure out if there's a bad configuration somewhere or if it's Apache directly, or if it's something wrong with vBulletin. It almost seems as if httpd gets thrown into a loop it can't get out and completely overloads itself. While watching top the system resources don't look horrible or a certain process using up everything. Sleeping tasks will rocket up to 300+ and at times I notice that kswapd is on top with 3% CPU.
It can't be that the server isn't able to handle the load because this new server is a huge upgrade. Athlon 4000+, 1GB RAM, 2x160GB SATA hard drives. I tried reinstalling Apache 1.3.33 and upgraded PHP to 4.4.6 and mySQL to 4.1.22 just yesterday. I'm completely lost and don't even know where to go from here. I'm tired of spending hours watching top waiting for it to overload. ANY help would be appreciated because this is getting bad.
I have ran across a thread on the CentOS website with people having the same issue because of the kernel, but it was a year old and would hope that the problem has been resolved. Thanks again everyone!!
I just got a new server with CentOS 4.4. It was running fine, no issues for a few weeks, and then I transferred a pretty large vBulletin website over to it (170 users are usually on at any given time) and that's when the issues started. What happens is the server will run fine, great load (usually in between .5 and 1.2) then at random the load peaks and basically everything comes to a halt within minutes. When I say peaks, I mean I have actually seen it at 200+. At first my httpd error_log would be filled with "[error] child process 32530 still did not exit, sending a SIGKILL". I was getting so tired of having to submit a reboot ticket to my provider to have them do a hard reboot so I made a little script that checks the load every minute, if it's above 10 it restarts httpd. Then takes about a minute for the load to come back down assuming the script ran in enough time for it to enter the command. Now that the script runs that message in error_log isn't there, just says that httpd was restarted. No other logs looks suspicous at all and don't help at all.
Now I can't figure out if there's a bad configuration somewhere or if it's Apache directly, or if it's something wrong with vBulletin. It almost seems as if httpd gets thrown into a loop it can't get out and completely overloads itself. While watching top the system resources don't look horrible or a certain process using up everything. Sleeping tasks will rocket up to 300+ and at times I notice that kswapd is on top with 3% CPU.
It can't be that the server isn't able to handle the load because this new server is a huge upgrade. Athlon 4000+, 1GB RAM, 2x160GB SATA hard drives. I tried reinstalling Apache 1.3.33 and upgraded PHP to 4.4.6 and mySQL to 4.1.22 just yesterday. I'm completely lost and don't even know where to go from here. I'm tired of spending hours watching top waiting for it to overload. ANY help would be appreciated because this is getting bad.
I have ran across a thread on the CentOS website with people having the same issue because of the kernel, but it was a year old and would hope that the problem has been resolved. Thanks again everyone!!
