Hacked, root commands run to change permissions and more..

WholesaleDialup

WholesaleDialup

Verified User
Joined
Sep 25, 2004
Messages
178
Location
San Antonio, TX
Today I tried logging into my DA system via Webmin but couldn't authenticate. Bad password etc..

So... Wow! Why couldn't I get in I thought.. I verified my password in my little password keeper app and it was correct. Also tried every other password it could be but.. Pretty odd since my browser remembers the password on my laptop and I never have to enter it if I am logging in from my laptop.

Finally had to assume that I had been hacked and they changed my password. ARRRRGG!!!

Luckily, I have SSH setup with a keypair so I don't need a password when logging in to SSH from my laptop. So, even though they changed the root password, I was still able to log-in as root and passwd to change the password back.

I don't allow root login via SSH without a key.

I don't allow any users SSH access on this server at all, just me.

I am running IP Tables.

So, I am sure I don't run the MOST secure box in the world but I have been at this for a number of years and I think it's pretty well secured. OK, Not Really since they got me!!

So, after changing the password for root to something new, I changed the DA password, my personal user account password and so on.

I started my research by hitting the up arrow and looking at the history and noticed they had run some pretty nasty commands as root. I also pulled the list from /root/.bash_history and here they are for reference:

id
rm -rf *
cd /etc/virtual
ls
exit
cd ..
rm -rf gh
ls -la
passwd
exit
ls -la
rm -rf *
ls -la
id
chmod 777 /home/acleorg756/domains
chmod 777 /home/*/domains/*/public_html
chmod 777 /home/*/domains/*
chmod 777 /home/*
chmod 777 /home/*/domains
passwd


Yikes, one thing I noticed right off is that it clearly shows where they ran passwd and changed my password for root. But when looking in the logs, I can't see any SSH connections except from my IP so I don't think they did it via SSH.

I am guessing they compromised the Joomla 1.5.20 install that lives under the acleorg756 user on this box. Just a guess but after all my research I think it's a pretty good guess.

I don't know what they actually deleted with the rm -rf commands and not sure how to tell.

They CHMOD 777 all the website files which I reset using the DA script that resets the permissions for those.

By the way I suspended the site with Joomla on it.

The reason I think they came through that site is because the first thing they did at the command line is "id" then a little later when they started chmoding the web files, they started with that directory first and I don't think it's a coincidence.

I also saw a FTP daemon exploit that came up on the exact day this started which is Nov 28th. I don't think it's related but I thought I would throw it out there.

So, at this point I am trying to determine what they actually did besides CHMOD my files, delete some files, changed my password and pissed me off.

I am not sure if I can trust the box anymore. I know the easy answer is to wipe it but it's a huge deal for me to do that because of a lack of backups and for other reasons.

If I can reasonably trust it then I really don't want to wipe it.

I am mostly brain dead and blind from looking at logs and forums etc. for the entire day. So, I hope some of this made sense.

I wanted to get this out here for two reasons:

1- So other having a similar issue can use the info to solve their issues.
2- Hoping someone out there will see something or think of something obvious that I haven't. Maybe some opinions on what may have happen etc.

Also, not sure what to do with the Joomla site. I researched vulnerabilities for that version and there was one but I didn't really find much info on it.

Finally, I am concerned that all the web files for 50-80 domains have been all set to 777 since Nov 28th until earlier today when I found and fixed. Was there more damage done through those wide open files?

Geesh, hackers really drive me nuts.. What is the point of this misery they cause.. :confused::eek::eek::eek::eek::eek:

Any help would be appreciated.
 
For security, you can change the SSH port .


About your issue, yes its weird but I've seen more weird stuff :)

I believe it's not a hack but just some general error caused due to the softwares you run. Since your password actually works so something maybe blocking the access otherwise.

Change the password and use it as it will overwrite the cause.
 
AsadMoeen said:
For security, you can change the SSH port .


About your issue, yes its weird but I've seen more weird stuff :)

I believe it's not a hack but just some general error caused due to the softwares you run. Since your password actually works so something maybe blocking the access otherwise.

Change the password and use it as it will overwrite the cause.
Click to expand...

I already run SSH on a non standard port.

My password didn't work until I logged in via SSH using a Keypair (not a password), and changed the root password back to something I knew.
 
Hard to say what they were up too.... but my first guess is using your box as a spam server.... I would keep an eye on your mail logs as all they might have wanted was to setup several accounts on your domains to use for sending spam.
 
Check PHP/HTML/CGI/.htaccess files on modification time (use something like find /home/ -mtime) in order to see if they're injected, just before google gives bans to your sites.

There can be redirects in .htaccess, or different JavaScript injections in files.
 
Usually, it is not that simple to get superuser privileges through a hole in PHP script (while you run it under user rights).

There must be something more critical. A security bug in software you're running on the server.
 
You wondered about what they might have removed with the rm -rf command. This would be probably the containings of your root homedir or the usershomedir they got shell access to.
However, before you reboot your server, check around if systemfiles are present.

If you run csf/lfd you should get a message about changed or missing operating system files.
 
You should also install and run chkrootkit (even if it's already installed, download it again); it'll check major programs/daemons to see if they've been compromised.

Generally if a hacker has obtained root we recommend rebuilding the server.

Jeff
 
Thanks to all for the suggestions, they are appreciated.

I will be working more on this and reading each of your suggestions carefully.

More than likely I will try to keep things going for a few days and start moving sites off to a new DA server in a new data center. I need to get things out of this data center anyhow for various reasons (mostly because they suck!).

Problem is of course that I hate to move the problem to the new server. If I move the sites to a new DA server, I probably move the hole unless I have identified it.

Even if I rebuild, once I restore any backups, I am just restoring the hole to allow them in again. :-(

None of it unfortunately is going to be a great solution unless I can find out how they did it in the first place but... You never can truly know, can you...
 
You never can truly know, can you...
Click to expand...
Not unless you happen to find it in some logs, and then spit out what possibility's they have or had after the hack.
But with a rootshell, you never know, that's why a new server install is the best solution.

However, if you could manage to find how they got in via some logs, you could choose not to restore that account. That way you get more chance they won't get in on the new server. Well.... with good security the chance is small. All servers can be hacked, there is no 100% security.
 
You must log in or register to reply here.
Back
Top