Weird issue with HTTPD stopping in the middle of the night and not restarting

I

idawgik

Verified User
Joined
Jun 19, 2009
Messages
12
Hello,

I'm having a weird issue with one of our servers and not really sure where to even begin looking troubleshooting it.

Basically, almost every night now, in the middle of the night, the httpd service will stop running. Trying to start it back up returns an error that port 80 is already in use. The only way we can get it back up is to restart the server.

Any suggestions?

Thanks
 
probably someone of your users has been hacked from a perl script or something like that and is binding port 80.

You need to check for strange process running and specially the one is binding port 80.

a netstat | grep :80 should help for start
 
Even better:

Code: 
netstat -npa | grep LISTEN | grep 80
 
Thanks for that, I'll run that next time it goes down (probably tonight if history is any indication).

For reference, here is what it shows right now when everything is running normally:

Code: 
tcp        0      0 209.31.96.180:53            0.0.0.0:*                   LISTEN      13166/named
tcp        0      0 0.0.0.0:21                  0.0.0.0:*                   LISTEN      8029/proftpd: (acce
tcp        0      0 :::80                       :::*                        LISTEN      7047/httpd
unix  2      [ ACC ]     STREAM     LISTENING     10808  5447/dovecot        /var/run/dovecot/dict-server
unix  2      [ ACC ]     STREAM     LISTENING     16307  8069/xfs            /tmp/.font-unix/fs7100
 
when you'll do the command with httpd not running if you till will have a :80 port used you need to check what command is running it (in your post for example command is 7047/httpd - where 7047 is the process id/pid).

then investigate where is that file (usually if a website has been hacked the file will be in /tmp/.SOMETHING)

for find that programm running you should use

find / -name PROCESSNAME

for see where is the executable.. prolly will be owned by apache:apache or something like that...

For furthere information someone should help you more telling you how investigate about how and what user has been hacekd.

Regards
 
Ok, finally caught it down again.

I can't start httpd, it says port 80 is already binded. Running that command though yeilds this:

Code: 
tcp        0      0 0.0.0.0:995                 0.0.0.0:*                   LISTEN      480/pop3-login
tcp        0      0 0.0.0.0:110                 0.0.0.0:*                   LISTEN      480/pop3-login
tcp        0      0 209.31.96.180:53            0.0.0.0:*                   LISTEN      11024/named
tcp        0      0 :::80                       :::*                        LISTEN      2191/perl
unix  2      [ ACC ]     STREAM     LISTENING     10809  5453/dovecot        /var/run/dovecot/dict-server

That seems to indicate that nothing is using it....?

Now I'm really confused...
 
Code: 
tcp        0      0 :::80                       :::*                        LISTEN      2191/perl

As i sayd is a perl script using that port, check the information about that pid 2191 and search for all .pl file with the command i gave you
 
idawgik said:
That seems to indicate that nothing is using it....?
Click to expand...

You are not blind are you. I think you mean it seems to indicated that something IS using it since something clearly is.

Seriously if you cannot see that then you need to stop what you are doing and immediately hire a server administrator to investigate your system and show you what is going on.
 
try:

Code: 
find / -name .pl

to find out where perl scripts are

check your /tmp /var/tmp directory for hidden files (. before the filname make the file hidden)

investigate for all directories and files with 777 permissions
 
Floyd: Sorry about that, that's what happens when I look at stuff after being woken up in the middle of the night and still half asleep.

I will take a look with the commands you guys provided to isolate the script.

Thanks again.
 
Ok, I managed to catch the server down again.

Perl is running, but I can't seem to find WHAT it's running.

I did the suggested find command looking for any .pl files. There are none.

Then I did some investigating into the process directory to try and find more info.

looking in /proc/#####/cmdline just shows /usr/bin/perl, no attributes with it.

Additionally, I looked in the /proc/#####/fd and all the files there were symlinks to the log files for the sites on the server.

I've been doing google searches to try and find information about what I can do.

Any suggestions on how to find out exactly what script is being run from an active process?

Thanks again.
 
You must log in or register to reply here.
Back
Top