Httpd not restarting. Possible hack?

mimeslav

New member
Joined
May 28, 2012
Messages
7
Hello, my problem is this:

At 2am my httpd service shut down almost every night (maybe a cron task?) and it didn't restart, only by rebooting the server.

I look into the error_log file and i found that someone is trying to do something in my server, and he connects thorugh port 80, so i think this attempt to "hack" my server is causing that httpd didn't restart because port 80 is occupied or something.

This is an example of the log i found these days and i don't know what to do to solve this problem. Any suggestions?

----------------------------
HTTP request sent, awaiting response... 211.49.162.173
Connecting to unyul.or.kr|211.49.162.173|:80... Can't open perl script "iso.jpg": No such file or directory
... .....200 OK
Length: 26191 (26K) [image/jpeg]
Saving to: `iso.jpg.1'

0K .Can't open perl script "iso.jpg.txt": No such file or directory
sh: curl: command not found
Can't open perl script "iso.jpg": No such file or directory
Can't open perl script "iso.jpg": No such file or directory
sh: fetch: command not found
sh: fetch: command not found
......Can't open perl script "iso.jpg": No such file or directory
------------------------------------

Thank you for your attention.
 
You should hire a security expert to look at your server. If they are able to bind to port 80 then they have root privileges already.
 
@scsi:

Where do you see that there's an intruder binding to port 80? What it looks like to me is someone trying to get a file from apache, which serves the public on port 80.

What am I missing?

Jeff
 
@scsi:

Where do you see that there's an intruder binding to port 80? What it looks like to me is someone trying to get a file from apache, which serves the public on port 80.

What am I missing?

Jeff
 
OP says that http can't start because 80 might be in use. He should find out if it's really the case; when you try to restart http, look carefully at the error it gives and check what's running at port 80 using a command like this:

Code:
netstat -nap | grep :80.*LISTEN
 
OP says that http can't start because 80 might be in use. He should find out if it's really the case; when you try to restart http, look carefully at the error it gives and check what's running at port 80 using a command like this:

Code:
netstat -nap | grep :80.*LISTEN

Thank you, next time I'll do what you say.

Last night the problem happened again at 3.35am but i reboot the server before I read this. In the log file other "intrusion" appears again, like all these days since the problem is happening and it always ends with this line when httpd stops:

[Wed May 30 03:35:02 2012] [notice] caught SIGTERM, shutting down

I've read something about making "graceful restart" editing httpd script or changing tmp folder permissions...any suggestions? In my tmp folder there's only several "sess_xxxxxx" files, should I remove them?

Again, thank you for helping me, i'm not an expert in these matters.
 
Today something new has ocurred. The problem is that httpd is consumming a lot of memory (a lot of pids in httpd status), so the webs we have on the server don't load althought httpd isn't stopped.

I have run the netstat command and only the httpd process was using port 80, then I stop httpd process, and when I tried to start it, that gave me this error:

(98)Address already in use: make_sock:could not bind to address [::]:80
(98)Address already in use: make_sock:could not bind to address [0.0.0.0]:80

So I reboot the server again...

Should I upgrade apache version to improve security and prevent these attacks to my server? A few months ago I upgrade php version and I have a lot of problems with the server webs ... is the apache upgrade safe?
 
You don't need to reboot the whole server to stop httpd completely.

You can first check if there are any httpd processes with:

Code:
ps -A | grep httpd

If there are one or more processes, you can kill them all by using killall:

Code:
killall -9 httpd

Then check again with the ps command to see that they're gone.

And then start httpd again normally.

To answer your question, yes you could upgrade to 2.4. Many people have it. But I'm not sure if its going to solve the problem, if you are under attack or just have too little memory.

In your earlier post, in what log file did you see

HTTP request sent, awaiting response
 
You don't need to reboot the whole server to stop httpd completely.

You can first check if there are any httpd processes with:

Code:
ps -A | grep httpd

If there are one or more processes, you can kill them all by using killall:

Code:
killall -9 httpd

Then check again with the ps command to see that they're gone.

And then start httpd again normally.

To answer your question, yes you could upgrade to 2.4. Many people have it. But I'm not sure if its going to solve the problem, if you are under attack or just have too little memory.

In your earlier post, in what log file did you see

Thank you for your help. The lines I posted earlier are in /var/log/httpd/error_log.

I will kill the processes next time instead of rebooting.

The problem with httpd started a week ago, the same time those errors and rare connections appeared in the error_log file, so i think server memory isn't the problem...The processes using httpd increase during the day.
 
That are indeed strange entries for a httpd error log, there's probably a malicious script somewhere. Try looking in /tmp or else under a user. Also if it was possible to run these kinds of scripts, 0day bugs might have been exploited so root access might also be at risk. I recommend letting an experienced system admin looking at your server (not me).
 
You don't need to reboot the whole server to stop httpd completely.

You can first check if there are any httpd processes with:

Code:
ps -A | grep httpd

If there are one or more processes, you can kill them all by using killall:

Code:
killall -9 httpd

Then check again with the ps command to see that they're gone.

And then start httpd again normally.

To answer your question, yes you could upgrade to 2.4. Many people have it. But I'm not sure if its going to solve the problem, if you are under attack or just have too little memory.

In your earlier post, in what log file did you see

When httpd is stopped there's no processes running httpd (using 'ps' command). So 'killall' command doesn't have anything to kill. 'httpd -k start' continues giving the same error:

(98)Address already in use: make_sock:could not bind to address [::]:80
(98)Address already in use: make_sock:could not bind to address [0.0.0.0]:80

Rebooting again...
 
Well if it's not httpd occupying port 80, try the netstat command again to see what is. It might give you an indication of what's going on.
 
Well if it's not httpd occupying port 80, try the netstat command again to see what is. It might give you an indication of what's going on.

Httpd stopped again and It seems like now the "exim" service is occuping port 80 and there are 22 processes running it (other days before httpd service had more than 30 processes)

I think a shellbot or something like that is overloading the server...
 
I think the problem is solved. I found a virus in one of our domains. There were a lot of files infected with "sheelbot" and "backdoor" viruses and many of them had the same name I saw in error_log file all these days since the problem started.

I cleaned the user, changed its password and httpd is now running without problems.

Thank you for your replies, now I know more things that I can check when my server crashes.
 
Back
Top