Some sites hacked - what to do now?

D

duncan

Verified User
Joined
Jan 19, 2005
Messages
93
Location
Halifax, NS, Canada
Most of the Joomla sites on my server were hacked this morning. Their index.php pages were all changed. Aside from a few other changes, they otherwise appear to remained untouched.

Oddly enough one other (non Joomla) site was affected - it's htaccess file was changed, but nothing else. The timestamp on this change appears to be after the other Joomla sites were changed. I believe the file had a chmod of 777.

I located a number of IPs that were sending SYN packets to the server - all have since been banned. I found them by running "netstat -ae | grep irc" and "netstat -ea | grep 666".

I also checked my cron log to see if there is any odd activity. All looked normal, although I am not sure if the following is a normal system command:
Code: 
Jan  4 05:34:01 vps crond[26429]: (apache) CMD (/var/tmp/.,/.img/update >/dev/null 2>&1)
Any idea if this is normal?

Aside from all of this - what else would you recommend I do? I will make sure all the Joomla sites are updated to the current release. But aside from that I'm not sure what else I can do to ensure the hole is closed and the site/content are secure.

Any help would be greatly appreciated!

Duncan
 
Last edited:
The main hole you've got in your system is that anyone in the world who knows how (even me) can change any file in your public_html paths that are chmod 777.

When we've seen this in the past it's generally been changes only to index.php files, but it's possible that other files are affected. Your best bet to discuss this issue is on a Joomla-specific forum.

You should probably start running PHP as cgi, instead of as an apache module, often these problems occur because Joomla sets all file ownership to apache, and anyone in the worl who knows how (yes, even me, again) can change any file in any of your public_html paths that are owned by apache.

Jeff
 
Hi Jeff, please explain in simple way what we can do to prevent you from hacking our site:)
 
jnet, just do what you've got to do to keep others out and you'll keep me out as well. If there are any backdoors into DirectAdmin, I don't know of them.

:D

Jeff
 
Jeff,

Thanks for the update. Is there any disadvantage to setting PHP to run as cgi? Is it a hard thing to change?

I have been really taken off-guard by this experience - I would never have thought that a simple 777 file could compromise the entire server. Maybe I had too much faith in the underlying CP and OS software. How do other hosts protect their systems from a user unknowingly uploading a 777 index.php file?

I'm afraid that this goes far beyond just defacing a few websites. A security specialist with our server provider did some looking, and had this to say:

I've reviewed your system to determine the scope of this compromise and found a number of processes impersonating as your webserver software. I've identified the components responsible for this and disabled the offending scripts and auto-start locations. For the most part, these were all stored in /var/tmp/.,/.img and /tmp/.aw. Both directories are disabled. The scripts within are typical IRC bot programs which connect out to a predefined IRC server and channel, which then send commands back to the infected system for remote command and control. There was a crontab entry setup as the apache user which was ensuring these programs would start if stopped or killed off, this too has been disabled.

I also noted while performing some of the cleanup, that there seems to be a script somewhere which is restarting the apache software if it's stopped. I had to stop the software in order to properly identify which processes were illegitimate and masquerading as the real software. Each time within a few minutes of the software stopping, it was restarted. I'm not certain if this is something you've designed to happen, or is the result of another process.

With regard to some of the other malicious software found, using chkrootkit I found two under /tmp/, httpd and httpd.1. Running strings on each resulted in a few lines which are hard to explain as anything but malicious 'backdoor', for example. chkrootkit identified these by their name, so it may not be particularly accurate, but it noted they were Slapper variants.

chkrootkit also found hidden processes not reported by standard system binaries and proc listings. This may be indicative of a kernel level rootkit, which if true, means you can't trust anything the system tells you. Unfortunately, proving this one way or another would require some time to take the system offline, perform a forensic analysis of its state and still then, it's not likely to be a definitive answer.
Click to expand...

What are your thoughts on this? Total reinstall of the box? :eek:
 
Last edited:
I also noted while performing some of the cleanup, that there seems to be a script somewhere which is restarting the apache software if it's stopped.
Click to expand...

DA does that for all monitored services.
 
duncan said:
Thanks for the update. Is there any disadvantage to setting PHP to run as cgi? Is it a hard thing to change?
Click to expand...
You can change it if you run (or change your system to run) custombuild instead of customapache. That's recommended.

However running PHP as CGI is definitely slower. Whether or not your users will notice that depends on your normal server loads. We've built one server with PHP running as CGI, and so far all is good.
I have been really taken off-guard by this experience - I would never have thought that a simple 777 file could compromise the entire server. Maybe I had too much faith in the underlying CP and OS software. How do other hosts protect their systems from a user unknowingly uploading a 777 index.php file?
Click to expand...
Some scripts and hosting platforms require files to be chmod 777. You should always have safe_mode turned on for all your domains. If your users complain that their scripts won't run with safe_mode turned on then you have the choice of not allowing the script (possibly losing the user), compromising your server, or switching to PHP as CGI.

You should also have Open BaseDir turned on, and Open BaseDir settings set to something reasonable so if a user is compromised it can't be spread to the entire system.

Note I'm NOT an expert in PHP security; hopefully smtalk will respond here as well; I always ask him my PHP security questions :).
What are your thoughts on this? Total reinstall of the box?
Click to expand...
Generally yes, but more importantly, you've already asked your resident expert to look, and he appears to recommend it as well, though he doesn't actually say so.

It's a learning experience :(.

Jeff
 
I have been really taken off-guard by this experience - I would never have thought that a simple 777 file could compromise the entire server.
Click to expand...

I believe there was more than one attack. A php script running as apache can only do so much damage.

However a vulnerable php script can allow an attacker to upload phpshell type script where the attacker can run other commands to further compromise the server.

So looking at the result I believe that either there were two separate attacks or one led to the other.

Either way rebuild the machine. Then run a script as a cron job to detect apache processes that are running that are not supposed to be running. I posted one in the forum somewhere.
 
Thanks both of you for the very helpful responses. This forum is a great resource.

It looks like I will be reinstalling the system:mad:. Jeff, I may end up contacting you for your (excellent) admin services with the re-install.

Cheers,

Duncan
 
You must log in or register to reply here.
Back
Top