Server hacked, now what?

[HMTKSteve]

I have been having some serious hacking problems on one of my domains. Long story short the main DA account for that domain was hacked. I was able to go through the logs and find what was altered and make repairs as well as change passwords and the like. The only way they could have gotten the password is via a brute force attack because I never use the main domain account (one of mine) as I either login to the domain while logged in as admin or create a temporary FTP account for file transfers.

However, while scanning the logs I found one successful login as root by the same IP that was infecting the targeted domain. The connections last about one second before they disconnect (according to the logs) and after I changed the passwords I noted a few fresh attempts to access the compromised accounts from the previously successful IP that failed.

According to the brute force monitor I have thousands of these "Attacks" coming in every day.

Now comes the part that is very troubling to me. DA rotates the logs so that I only have 30 days of logs to go through. I hope that I caught the root login early enough that it was just a lucky shot or some automated hack just checking to confirm the password was still good but with only 30 days of logs I don't know.

A hacker getting root access could certainly explain the weird problem I had earlier with bind running as root. Equally troubling is that the compromised domain account was running something called "kernelupdates" a few days ago with a very high server load. I could not find anything on my server with that name nor did Google help me.

Some of the domains on my server are large with hundreds of thousands if not millions of files. Any backups I have are likely compromised since I have no idea when the first attack succeeded.

What can I do?

EDIT: Using last I find hundreds of zero time root logins from a Russian IP address starting three weeks ago. Logins show zero time and are time 30 minutes apart. My guess is they were automated and designed to login, run their script and logout. No unknown logins since I changed passwords.

 

[sysdev]

The real question is.... do you and your customers feel happy knowing the server was actually rooted?

Are you sure nobody logged in as root and cleaned the logs file, leaving only the automated registrations in the logs? Maybe, hooping you would think like you think now and most likely are not reinstalling your server?

Wat you can and should do, is reinstall a new server/vps, secure it and install all virusscanning, spam, firewall gadgets you need. Make sure you use mpm_itk (apache module) or mod_ruid2 and restore your 'best' backups after you scanned them with a virus- and malware scanner.

Finally (after praying intensively to your favorate god and hoping for the best), at least disable root logins and use ssh-keys.

Good luck!

 

[HMTKSteve]

How can I trust anything that I take off of the server? Everything would be suspect.

If I were to copy everything from /home, reformat and reinstall what can I use to verify the integrity of the files before I put them back up?

 

[zEitEr]

Hello,

If you run a new server and set secure permissions, disable dangerous PHP functions, enable firewall and scan your home files as it was already suggested you are quite safe to restore sites (not binaries) from backups. Even if you still have malware in public directories of your accounts, they won't be able to get root access to your server.

 

[toml]

Another thing too you should always do, is run something like logwatch and scan the reports every day. It does a good job of summarizing the activity on the server for that past day, and it would tell you exactly how many times someone successfully logged into the server, what account and from where as well as how many failures there were.

As was mentioned never enable root ssh logins, with the proliferation of distributed brute force attacks out there it is only a matter of time for someone to get in, at least if they compromise a non-root account they can only do minimal damage and still have to find a way to get root permissions, which you should be able to catch before they can do damage by monitoring the log files.

 

[HMTKSteve]

It looks like a computer that I used to connect to my server was compromised.

I have run RKHunter and maldet and have thus far come up clean regarding major problems. I also changed the ports for FTP and SSH as well as removing roots ability to login via SSH.

I backed up all domains and downloaded copies to my local machine. I then checked the backups for malware and found one other compromised account.

I also used a clean computer to change the root password (again) and create a new account just for SSH and sudo.

If I change the admin password for directadmin do I have to do anything to insure directadmin still works correctly? I seem to recall the admin password being stored in the filesystem?

 

[HMTKSteve]

One thing I do not understand is if they had acquired root access why didn't they change the root password and lock me out?

 

[nobaloney]

One thing I do not understand is if they had acquired root access why didn't they change the root password and lock me out?

Because they don';t want you to catch on. They may be using the server to do something.

If they've logged in as root (even only once) they could have easily done a root level hack and you can no longer trust the logs, the directory lists, or a lot of other things.

In my opinion, zEIter has the only reasonable answer. Rebuild from scritch.

And of course run rkhunter immediately upon building the system so it has a clean system to compare against for the future.

Jeff