So, yep, I finally got hacked

TomJones

Verified User
Joined
May 9, 2004
Messages
59
My own fault, I let several customers go far too long without updating their joomla CMS. Three of them had 1.5.26 installed, which was the last in the 1.5 platform (and ancient by now). Two days ago, one of the sites started sending out tons of the "Your Package was Not Delivered"-type spam emails with a malicious link or zip attachment through Local Relay. They got around 100,000 out before I disabled the user. First thing yesterday I started looking into all three of the sites and found what you never want to find. Two were exploited using a fairly recently discovered vulnerability in ext_explorer, both were hacked on December 21st. I removed that extension and found all the files created between then & now and deleted them.

The third, though, was hacked back in April of last year. It had been hacked through a vulnerability in TinyMCE on 1.5 and oh man, it was it a mess. I found at least a dozen files on the site that variations on eval(gzinflate(base64_decode( and the encoded strings were a page+ long. The one file that scared me the most had "Web Shell by oRb" commented at the top of the file. So pretty sure I can't trust anything on that user anymore. I changed file names on some (so they were no longer php files) and deleted others. Hopefully I cleared most of the problems, I am going to delete their entire site and start fresh, but it's going to take some time to migrate each site to the 2.5 platform.

All this is not to just publicly humiliate myself, I'm hoping someone can tell me know bad the situation on my server is. I am running the latest/last CustomBuild on a Centos6 64bit box with the mod_ruid module for apache. I've checked in /etc/passwd, doesn't seem like any new users have been created. With PHP running as the user, how deeply can they penetrate into the system? Is completely wipe those user accounts and starting over enough? I've also just installed the Linux Malware Detect tool, it's doing it's first scan now.

Anybody with more knowledge on these types of problems have any advice on what steps I should take?
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,788
Location
A Coruña, Spain
As far as i can understand your system should be fine, mod_ruid2 should ha protected you enough for this.

If you can confirm that suspend those user stop outgoing spam email so your pretty fine and lucky, so, wipe those users (or just remove the site/public_html folder) would be enough.

A Malware Detect scan is pretty nice, also a ClamAV and RKHunter scans would be highly suggested.

Does you notice any other issue except for those outgoing spam email?
Have you found which file is corrupted? (Using php_mail.log present in .php of each user account)

Regards
 

TomJones

Verified User
Joined
May 9, 2004
Messages
59
Yep, disabling the user immediately stopped the spam. I have ClamAV and RKHunter installed. LMD didn't detect any problems, neither did RKHunter or ClamAV scans. I found the spam scripts pretty easily greping for create_function & eval. These search lines were most helpful in finding hacked files:
grep -rC 2 --include=*.php --exclude-dir=/home/admin "create_function" /home --color
grep -rC 2 --include=*.php --exclude-dir=/home/admin "eval(" /home --color
find . -type f -iname "*.php" -mtime -15 | xargs -r ls -l

In my frantic reading up on the subject yesterday, I noticed that some script techniques hide the names eval, base64_decode, gzipinflate, etc., through some clever techniques. I'm not completely convinced I got everything, but I reason that, if they had access to those better methods, they would have used them on all the files.

I haven't noticed any other issues. My underlying concern was the scripts installed back in April had turned my server into a zombie bot server or something. It seems like they successfully did everything they wanted inside that user account, hard to believe they didn't do anything more malicious with it. But, maybe they realized they were stuck executing scripts as the user and gave up....
 

Arieh

Verified User
Joined
May 27, 2008
Messages
1,198
Location
The Netherlands
Sounds like you have a lot of protection already. The only thing that I could think of is the number of e-mails that have been sent. You could put a limit (http://help.directadmin.com/item.php?id=81) to limit the damage. And/or you could use something like CSF, it can keep track of e-mails being sent, they call it LOCALRELAY_ALERT, as soon as many e-mails are being sent, you will be notified.

Other than that, the eval/base64 stuff sounds something typical to be picked up by LMD. Personally I don't use it, but does it update its rules (through a cron I believe)? And is it running correctly?
 

TomJones

Verified User
Joined
May 9, 2004
Messages
59
Restricting user emails is a great suggestion, I had seen that setting option in the Administrator Settings, but never set it (would have saved me the time I'm spending getting delisted from the spam blacklists). Set it to 1000, that should be safe. It was CSF's LOCALRELAY_ALERT that alerted me to the problem, I just wasn't checking my server email account during the day on Sunday. I need to customize CSF better so I can forward the alerts to my personal email. As it is now, I get so many port scan and failed login attempt emails from CSF that it would flood my inbox.

LMD is installed and seems to be working properly. Scan ran & didn't find anything, so hard to say with 100% confidence. I followed these install instructions to get the LMD CRON set-up properly for DA:
https://forum.directadmin.com/showthread.php?t=45851
 

massive

Verified User
Joined
Feb 16, 2006
Messages
298
Yep, disabling the user immediately stopped the spam. I have ClamAV and RKHunter installed. LMD didn't detect any problems, neither did RKHunter or ClamAV scans. I found the spam scripts pretty easily greping for create_function & eval. These search lines were most helpful in finding hacked files:
grep -rC 2 --include=*.php --exclude-dir=/home/admin "create_function" /home --color
grep -rC 2 --include=*.php --exclude-dir=/home/admin "eval(" /home --color
find . -type f -iname "*.php" -mtime -15 | xargs -r ls -l

In my frantic reading up on the subject yesterday, I noticed that some script techniques hide the names eval, base64_decode, gzipinflate, etc., through some clever techniques. I'm not completely convinced I got everything, but I reason that, if they had access to those better methods, they would have used them on all the files.

I haven't noticed any other issues. My underlying concern was the scripts installed back in April had turned my server into a zombie bot server or something. It seems like they successfully did everything they wanted inside that user account, hard to believe they didn't do anything more malicious with it. But, maybe they realized they were stuck executing scripts as the user and gave up....
Also a good option is to install maldet . It uses the clamav binary and has option to automaticaly move the infected files into different directory
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
@TomJones:

It appears you're (belatedly) doing everything right. I used to use a limit of 200 daily emails for users, but on new servers we're setting a default outgoing email limit of 1000. That, an account with the AOL postmaster so they'll automatically notify you of every email reported as spam by every AOL users, should protect your email (just read all those AOL SCOMP emils and follow up on th suspicious ones).

Note to others: RKHunter won't help you unless it has a baseline for your server; install it and run it NOW, while you know your server is clean.

Jeff
 
Top