httpd does not start

entit · Apr 20, 2009

Hello,

The httpd service can not start in DA!
So I run /sbin/service httpd start 2>&1
It showed:
Starting httpd: [Mon Apr 20 17:00:34 2009] [warn] module php5_module is already loaded, skipping
Warning: DocumentRoot [/home/entitres/domains/suspended/mail] does not exist
Warning: DocumentRoot [/home/entitres/domains/suspended/mail] does not exist
Warning: DocumentRoot [/home/entitres/domains/suspended/mail] does not exist
Warning: DocumentRoot [/home/entitres/domains/suspended/mail] does not exist
Warning: DocumentRoot [/home/vietransgn/domains/vietransaigon.com/public_html/304x] does not exist
(98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

Please help me to fix this error!

scsi · Apr 20, 2009

lsof -i | grep 80

netstat -na | grep -i list | grep 80

tillo · Apr 20, 2009

You can read a possible reason and solutions here (ignore any reference to FreeBSD).

entit · Apr 20, 2009

Hello scsi,

Nothing is listen on port 80!

[root@server1 ~]# netstat -na | grep -i list | grep 80
tcp 0 0 :::80 :::* LISTEN
unix 2 [ ACC ] STREAM LISTENING 8084 /tmp/.font-unix/fs7100

Please advise me, how to check and fix this error!
Thanks!

tillo · Apr 20, 2009

Yes, from your last post I see something listening.
Try with "netstat -pln |grep 80" or "lsof -i TCP:80 |grep LISTEN", it will show you the blocking process.

entit · Apr 20, 2009

This is the output:

[root@server1 ~]# lsof -i TCP:80 | grep LISTEN
perl 11440 apache 4u IPv6 17443289 TCP *:http (LISTEN)
perl 11446 apache 4u IPv6 17443289 TCP *:http (LISTEN)
[root@server1 ~]# netstat -pln |grep 80
tcp 0 0 :::80 :::* LISTEN 11440/httpdse
unix 2 [ ACC ] STREAM LISTENING 8084 2651/xfs /tmp/.font-unix/fs7100

tillo · Apr 20, 2009

Well, you just discovered that not only you have the mod_php problem I was talking about in the earliest post, but your server also catched a worm.

This is what I would do if I were you:

check where that perl script (the worm) resides, by running:
Code:
```
find /tmp /var/tmp /dev/shm /home/tmp -name "*se"
```
(the worm tries to hide himself as "httpd" but its name is too long, and ends in "se")
if the previous command doesn't show anything, try with:
Code:
```
grep -RI httpd /tmp /var/tmp /dev/shm /home/tmp 2>/dev/null
```
if the previous command doesn't show anything either, try with:
Code:
```
lsof -p 11440 |grep -e tmp/ -e shm/
```
if something was found, copy it somewhere secure and remove it from there
kill the worm by running:
Code:
```
kill -9 11440
```
find which web software had the flaw that allowed that worm to enter your system (it was probably roundcube, check in /var/www/html that you only have roundcube 0.2) and patch/upgrade/remove it
restart Apache

entit · Apr 20, 2009

[root@server1 ~]# lsof -i TCP:80 | grep LISTEN
perl 11440 apache 4u IPv6 17443289 TCP *:http (LISTEN)
perl 11446 apache 4u IPv6 17443289 TCP *:http (LISTEN)
[root@server1 ~]# netstat -pln |grep 80
tcp 0 0 :::80 :::* LISTEN 11440/httpdse
unix 2 [ ACC ] STREAM LISTENING 8084 2651/xfs /tmp/.font-unix/fs7100

[root@server1 ~]# find /tmp /var/tmp /dev/shm /home/tmp -name "*se"

[root@server1 ~]# grep -RI httpd /tmp /var/tmp /dev/shm /home/tmp 2>/dev/null
/tmp/blue:#system("kill -9 `ps ax |grep httpdse |grep -v grep|awk '{print $1;}'`");
/tmp/blue:my $processo = 'httpdse';
/tmp/blue.1:#system("kill -9 `ps ax |grep httpdse |grep -v grep|awk '{print $1;}'`");
/tmp/blue.1:my $processo = 'httpdse';

[root@server1 ~]# lsof -p 11440 |grep -e tmp/ -e shm/
perl 11440 apache 593u REG 253,3 0 29556799 /tmp/ZCUDCy3SSO (deleted)

[root@server1 ~]# kill -9 11440

[root@server1 ~]# /sbin/server httpd start
-bash: /sbin/server: No such file or directory
[root@server1 ~]# /sbin/service httpd start
Starting httpd: [Tue Apr 21 06:42:40 2009] [warn] module php5_module is already loaded, skipping
Warning: DocumentRoot [/home/entitres/domains/suspended/mail] does not exist
Warning: DocumentRoot [/home/entitres/domains/suspended/mail] does not exist
Warning: DocumentRoot [/home/entitres/domains/suspended/mail] does not exist
Warning: DocumentRoot [/home/entitres/domains/suspended/mail] does not exist
Warning: DocumentRoot [/home/entitres/domains/suspended/mail] does not exist
Warning: DocumentRoot [/home/entitres/domains/suspended/mail] does not exist
Warning: DocumentRoot [/home/entitres/domains/suspended/www1] does not exist
Warning: DocumentRoot [/home/entitres/domains/suspended/www1] does not exist
Warning: DocumentRoot [/home/vietransgn/domains/vietransaigon.com/public_html/304x] does not exist
(98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

tillo · Apr 20, 2009

The worm is "/tmp/blue", and there is a copy called "/tmp/blue.1". Save a copy somewhere for further investigation then remove them both.
Next thing, search for that bugged software as I told you to.
A fast search on Google told me that "httpdse" is a known worm for Joomla, and especially for a bugged version of the perForms extension. Search your server for "perform.php" and remove it, then notify the customer to upgrade his Joomla distribution and every extension regularly.
Then, set register_globals to Off in your php.ini file.
Finally, restart Apache.

entit · Apr 20, 2009

Thanks Tillo,
As your guide, i have fixed!

bengrubb · Jun 28, 2009

Hey,

Edit: It appears to be working again...but very slowly.

I have an issue that is very similar.

The httpd service has stopped running.

I try to start is:

DS4407:/# /etc/init.d/httpd start
Starting httpd: Warning: DocumentRoot [/home/reseller/domains/suspended/forum] does not exist
Warning: DocumentRoot [/home/reseller/domains/suspended/kdr] does not exist
Warning: DocumentRoot [/home/user/domains/otherwi.se/public_html/folder] does not exist
Warning: DocumentRoot [/home/user/domains/domain_name/private_html] does not exist
[Mon Jun 29 00:32:18 2009] [warn] NameVirtualHost IP.IP.IP.IP:443 has no VirtualHosts
(98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

I run netstat:

DS4407:/# netstat -pln |grep 80
tcp6 0 0 :::80 :::* LISTEN 8492/httpd
DS4407:/#

I kill:

kill -9 8492

And it re-spawns:

DS4407:/# netstat -pln |grep 80
tcp6 0 0 :::80 :::* LISTEN 8493/httpd

Any ideas?

tillo · Jun 28, 2009

My guess is that you already have an Apache instance running, but its lock/pid file doesn't match the one from your Apache httpd init script. It happens for example if you run Apache with "apachectl" or directly with /usr/sbin/httpd instead of using the init script.

To be sure that a process is what he claims to be, run "ls -l /proc/<PID>/exe" and see if the symlink destination matches. If it doesn't (for example it's /usr/bin/perl or /usr/local/bin/php or /tmp/whatever) you got yourself a worm or trojan.

bengrubb · Jun 28, 2009

tillo said:
My guess is that you already have an Apache instance running, but its lock/pid file doesn't match the one from your Apache httpd init script. It happens for example if you run Apache with "apachectl" or directly with /usr/sbin/httpd instead of using the init script.

To be sure that a process is what he claims to be, run "ls -l /proc/<PID>/exe" and see if the symlink destination matches. If it doesn't (for example it's /usr/bin/perl or /usr/local/bin/php or /tmp/whatever) you got yourself a worm or trojan.

Hey man,

It shows me the following:

DS4407:~# netstat -pln |grep 80
tcp6 0 0 :::80 :::* LISTEN 8493/httpd
DS4407:~# ls -l /proc/8494/exe
lrwxrwxrwx 1 root root 0 2009-06-29 01:22 /proc/8494/exe -> /usr/sbin/httpd
DS4407:~# ls -l /proc/8493/exe
lrwxrwxrwx 1 root root 0 2009-06-29 01:15 /proc/8493/exe -> /usr/sbin/httpd

It seems to be ok, right?

tillo · Jun 28, 2009

Yes, it is.

bengrubb · Jun 28, 2009

tillo said:
Yes, it is.

I also had this, but I killed the process.

DS4407:~# netstat -pln |grep 80
tcp6 0 0 :::80 :::* LISTEN 4666/exim

Why would "exim" be using port 80?

tillo · Jun 28, 2009

That's a known problem caused by both Apache and PHP developers bitching about who should act and make the other proud.

Read a description here: http://www.directadmin.com/forum/showthread.php?t=12232&p=145030#post145030

And for solution:

find the "restart)" case within your Apache init script (/etc/init.d/httpd)

put this just before the stop call:

Code:

httpd_childs=$(pstree -p $(ps u -C httpd |grep ^root |awk '{print $2}') 2>/dev/null |egrep -o '\([[:digit:]]+\)' |tr -d '()')

put this just before the start call:

Code:

for pid in $httpd_childs; do kill -9 $pid 2>/dev/null; done
listening_procs=$(netstat -pln |grep ':80 ' |tr / ' ' |awk '{print $7}' |sort -u)
for pid in $listening_procs; do kill -9 $pid 2>/dev/null; done

the result should be something like:

Code:

  restart)
	httpd_childs=$(pstree -p $(ps u -C httpd |grep ^root |awk '{print $2}') 2>/dev/null |egrep -o '\([[:digit:]]+\)' |tr -d '()')
	stop
	waitforexit "httpd" 20
	for pid in $httpd_childs; do kill -9 $pid 2>/dev/null; done
	listening_procs=$(netstat -pln |grep ':80 ' |tr / ' ' |awk '{print $7}' |sort -u)
	for pid in $listening_procs; do kill -9 $pid 2>/dev/null; done
	start
	;;

It's two workarounds in one, that can be avoided by using PHP-CGI instead of mod_php.

httpd does not start

entit

Verified User

scsi

Verified User

tillo

Verified User

entit

Verified User

tillo

Verified User

entit

Verified User

tillo

Verified User

entit

Verified User

tillo

Verified User

entit

Verified User

bengrubb

Verified User

tillo

Verified User

bengrubb

Verified User

tillo

Verified User

bengrubb

Verified User

tillo

Verified User