Apache dies 2nd night in a row

I

i2iweb

Verified User
Joined
Feb 26, 2004
Messages
41
Location
Dallas, TX
Hi,

Wanted to see if anyone can point me in the right direction. This is the second night that apache went down at the same exact time. I used portupgrade to upgrade a few things about 3 days ago which most likely upgraded perl in the process. A simple restart of Apache fixes the issue but how can I fix it for good? Below is an excerpt from my error_log.

System: FreeBSD 4.9
Current Perl Version: 5.8.2


Can't locate Cwd.pm in @INC (@INC contains: /usr/local/lib/perl5/site_perl/5.8.0/mach /usr/local/lib/perl5/site_perl/5.8.0 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.0/BSDPAN /usr/local/lib/perl5/5.8.0/mach /usr/local/lib/perl5/5.8.0 .) at (eval 2) line 1.
Can't locate Cwd.pm in @INC (@INC contains: /usr/local/lib/perl5/site_perl/5.8.0/mach /usr/local/lib/perl5/site_perl/5.8.0 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.0/BSDPAN /usr/local/lib/perl5/5.8.0/mach /usr/local/lib/perl5/5.8.0 .) at (eval 2) line 1.
Can't locate Cwd.pm in @INC (@INC contains: /usr/local/lib/perl5/site_perl/5.8.0/mach /usr/local/lib/perl5/site_perl/5.8.0 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.0/BSDPAN /usr/local/lib/perl5/5.8.0/mach /usr/local/lib/perl5/5.8.0 .) at (eval 2) line 1.
Can't locate Cwd.pm in @INC (@INC contains: /usr/local/lib/perl5/site_perl/5.8.0/mach /usr/local/lib/perl5/site_perl/5.8.0 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.0/BSDPAN /usr/local/lib/perl5/5.8.0/mach /usr/local/lib/perl5/5.8.0 .) at (eval 2) line 1.
 
Can't locate Cwd.pm in @INC (@INC contains: /usr/local/lib/perl5/site_perl/5.8.0/mach /usr/local/lib/perl5/site_perl/5.8.0 /u
sr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.0/BSDPAN /usr/local/lib/perl5/5.8.0/mach /usr/local/lib/perl5/5.8.0 .)
at (eval 2) line 1.

Same thing and I am running FreeBSD 5.1. Seems apache is dying every night some time between midnight and 12:31.
 
Yep, Mine dies at 12:31 as well so it looks like something running in cron kills it at that time. I'll do a little more research and let you know what I come up with. I know one thing that can be done is perl -c between 12:00 to 12:30 to see what's killing it. For the meantime I am trying to symlink the new perl to the old one to see if it helps.
 
Try installing Cwd:

Code: 
perl -MCPAN -e shell
cpan> install Cwd

Chris
 
Well, would this mean that something in the install process did not work and this should have already been installed?
 
I dont think DA even installs perl..

Please do not install services such as Apache, PHP, MySQL, Ftp, Sendmail, etc., as we will do this for you. All we need is a clean install of Redhat.
Click to expand...

In the packages directory:

da_exim-4.24-1.i386.rpm
MySQL-client-4.0.16-0.i386.rpm
da_vm-pop3d-1.1.7e-1.i386.rpm
MySQL-devel-4.0.16-0.i386.rpm
squirrelmail-1.4.2.tar.gz
imapd
MySQL-server-4.0.16-0.i386.rpm
webalizer-2.01_10-11.i386.rpm
Mail-SpamAssassin-2.55.tar.gz
phpMyAdmin-2.5.4-php.tar.gz
webmail.tar.gz
proftpd-1.2.9-1.i386.rpm
majordomo-1.94.5.tar.gz
proftpd-standalone-1.2.9-1.i386.rpm

Unless DA installs them if they are not there? There are not any perl rpms from DA on the system im checking here anyway... and perl is installed via rpm on this... (obviously FBSD doesnt use rpm BUT.... if it doesnt install on redhat its doubtful it installs on FBSD either)

Will leave Mark or John to confirm this....

Chris
 
DA doesn't touch perl. It will install it with pkg_add if there is *nothing* on the system.. (FreeBSD only).. but doesn't do any updgrades or module installs.

I'm currently looking into a possible cause with apache log rotation where the logs are removed during tally, and recreated when apache restarts. The issue is that the time between the deletion and restart can be several minutes so apache is writing to a non-existant file. That may or may not be releated to the problem at hand, but I thought I'd mention it in case it has some relevance.

John
 
DirectAdmin Support said:
The issue is that the time between the deletion and restart can be several minutes so apache is writing to a non-existant file.
Click to expand...

Would LogRotate (rather than the DA app) remove this problem and also allow the restart removed every night..... thats another thing I really dislike at the moment (besides the non live tally :p )

Or... another idea... instaed of:

rm -f log
then create the log

(assuming thats how its done now...)

Couldn't you make the copy then

echo " " > log

?

Chris
 
It is a little strange because I had DA installed on basically the same server setup and this did not happen.

Brad
 
The reason apache is dying (maybe) is beacuase it holds the error logs open from startup. For all other logs, it's not an issue because they're opened and appended each time, so if the log doesn't exist, it's created. Since the filedescriptor is held open, you should in theory be able to rename the log file and it would still fill up the renamed file because the filedescriptor just points to the node in the filesystem, and isn't dependant on the name once it's open. By deleteing the node, apache would need to reopen the filedescriptor in order to write to a pipe that has another end, but that isn't done until the tally is complete.

By just creating a new file, the node is different from the one apache is using, so it won't be used. I may be way off on this, but I'll change how it's done anyway so we can rule it out if it's not the cause.

It's possible that different OS's handle the dangling filedescriptor differently, but I'm not too sure one that.

John
 
Yea, I realise that the kernel wont remove an active file until the application, in this case apache, has finished with it - is that the reason to apache being restarted?

Chris
 
Well Apache crashed once more this time about 20 minutes later than usual. Any other thoughts on what may be happening?
 
ProWebUK said:
Would LogRotate (rather than the DA app) remove this problem and also allow the restart removed every night..... thats another thing I really dislike at the moment
Click to expand...

I would like to say - I would like to get rid of this restart too. It causes me headaches. Once or twice a week it does not restart properly, and it stays down until I notice it in the morning.

Can we get DA up to Apache 2 and use piped logs instead?

Please, please get rid of this restart...

John
 
Can all affected users please post their Operating Systems?

Is there *any* information in the apache logs as to why it goes down?

I've changed the code to trunctate the logs to zero which apache doesn't seem to mind at all. (instead of deleting them)

The restart is done when the httpd.conf files are rewritten.. For example if a user gets suspended, his DocumentRoot changes, and a reload is needed. I don't think having the restart is the problem, I think the reason the restart isnt working is the problem (restarts are done all the time.. when new domains are added, ssl certs, subdomains.. etc)

John
 
RedHat 9.0

From my post in another thread:

TurtleBay said:
No, not Zend Optimizer. When I tried to install it last time, it made Apache segfault. I have not tried to install it lately.

Log file looks like this:

[Tue Mar 9 00:15:02 2004] [warn] child process 30162 still did not exit, sending a SIGTERM
[Tue Mar 9 00:15:05 2004] [error] Cannot remove module mod_frontpage.c: not found in module list
[Tue Mar 9 00:15:05 2004] [crit] (98)Address already in use: make_sock: could not bind to port 443

Apache then sits in memory, but not responding to anything.
Click to expand...

Happens once or twice a week.

John
 
Can't locate Cwd.pm in @INC (@INC contains: /usr/local/lib/perl5/site_perl/5.8.0/mach /usr/local/lib/perl5/site_perl/5.8.0 /u
sr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.0/BSDPAN /usr/local/lib/perl5/5.8.0/mach /usr/local/lib/perl5/5.8.0 .)
at (eval 2) line 1.

FreeBSD 5.1

That is really the only information I could find. Nothing at all in /var/log/messages. If there is anywhere else I can look, let me know.
 
[Tue Mar 9 00:15:02 2004] [warn] child process 30162 still did not exit, sending a SIGTERM
[Tue Mar 9 00:15:05 2004] [error] Cannot remove module mod_frontpage.c: not found in module list
[Tue Mar 9 00:15:05 2004] [crit] (98)Address already in use: make_sock: could not bind to port 443
Click to expand...
It Could also be possible that a script run by apache is hanging around during the time of the restart.

Another thing to try would be to put 1 or 2 seconds of "sleep" between the stop and start in the apache boot script during a restart.

John
 
DirectAdmin Support said:
It Could also be possible that a script run by apache is hanging around during the time of the restart.

Another thing to try would be to put 1 or 2 seconds of "sleep" between the stop and start in the apache boot script during a restart.

John
Click to expand...

John,

How would I test for the first possibliity, and how do I implement the second?

Thanks,
John
 
You'd need to do a "ps -ax" and look for a script, probably running as a user right when apache refuses to restart. Just cross check that with any scripts your users might have installed (could take a while).. and then see if the script is buggy.

The sleep can be implemented by editing the apache boot script:
Redhat: /etc/init.d/httpd
FreeBSD: /usr/local/etc/rc.d/httpd

Scroll down to the part that says "restart". It should have "stop" and "start" after each other. Insert a line in between those 2 commands and enter:

sleep 2

to give it a 2 second pause after it stops before starting up again.

John
 
DirectAdmin Support said:
You'd need to do a "ps -ax" and look for a script, probably running as a user right when apache refuses to restart. Just cross check that with any scripts your users might have installed (could take a while).. and then see if the script is buggy.

The sleep can be implemented by editing the apache boot script:
Redhat: /etc/init.d/httpd
FreeBSD: /usr/local/etc/rc.d/httpd

Scroll down to the part that says "restart". It should have "stop" and "start" after each other. Insert a line in between those 2 commands and enter:

sleep 2

to give it a 2 second pause after it stops before starting up again.

John
Click to expand...

The server that's doing it doesn't have anyone on it yet. I just added the command and hopefully everything goes well tonight.
 
You must log in or register to reply here.
Back
Top