help - server hacked

T

tomc

Verified User
Joined
Feb 22, 2006
Messages
25
Last night one of our servers was hacked. We shutdown the server instantly as we were aware the hacker was connected to ssh at the time.
There are some things we could have done to prevent the hacker logging into ssh (ie change the port, or not allowing root access from ssh), but there is one major thing we couldnt have stoped - and I'd like to know what it was / how it happened.

Around 30 seconds before the user logged into ssh as root (got the password correct first time), the root password was changed (not by me !).

Now the thing I'd like to know is - how ! how can a user with no ssh access change the root password !

They also have no physical access to the server.

They also dont know the root password (I assume this is the case otherwise they wouldnt have needed to change it before logging in).

The server is running CentOS, php is not in safemode, perl etc is enabled as the default DA install.

We have APF/BFD so its not a brute force attack.

Any ideas ?


Thanks

Tom
 
several possibilities
- you have a trojan which steels password (current) on your comp
- not php safe mode : hum why not? check all scripts, applications and versions because some can run with root privilege if any security bugs
- check all version installed on DA if up2date
- check people you give root/ssh access or give password, even on old servers if the password is same.

I hope you installed server from empty one your self, and you have a serious datacenter, does it know your password ? and do you know your users.

possible i miss some scenaries
 
tomc said:
Last night one of our servers was hacked. We shutdown the server instantly as we were aware the hacker was connected to ssh at the time.
There are some things we could have done to prevent the hacker logging into ssh (ie change the port, or not allowing root access from ssh),
Click to expand...


It doesnt matter whether you change the port or not if he has root he can log into the box on any port he wants especially if he has rooted the box. You dont need to have ssh turned on or off for users either. If he has opened his own backdoor he can ssh to the box whether your allowing it or not.

but there is one major thing we couldnt have stoped - and I'd like to know what it was / how it happened.
Click to expand...

You will never know unless you do the analysis yourself. Probably got in though an insecure script running on a website. All he has to do is construct a url and dump some scripts into the /tmp directory and his in. Really, unless you know Unix inside out i recommend that you contact a consulting company to do an analysis of your box and find the point of entry. I can recommend a company who is very reasonable and they will even clean the box if its saveable.

Around 30 seconds before the user logged into ssh as root (got the password correct first time), the root password was changed (not by me !).
Click to expand...

That doesnt work. Doesnt matter if you change the pw. If he has an ssh backdoor installed changing the root pw is fruitless

Now the thing I'd like to know is - how ! how can a user with no ssh access change the root password !
Click to expand...

See above
 
Thanks for the advice.
I am in the middle of reinstalling things, and will restore users from the last backup before the problem.
I have had a look around in the logs and things, it appears there are a few things it could have been - all related to it being php/perl that has been run. There are several scripts in one of the users folders that have hacker-like scripts in them and googling brings up results such as the "Crash_over_ride" backdoor.

How can I stop this (and other backdoors!) in future - I regularly run updates, but its impossible to stop someone uploading something that will try and "attack" the server. I dont want to restrict users too much (ie safemode) as this causes lots of problems. We also have many scripts on the servers that use more powerfull stuff (such as cacti, rddtool and there are many websites running calls to imagemagick and other command line tools) But obously I dont want them to be able to change the root password !!
 
tomc said:
Thanks for the advice.
I am in the middle of reinstalling things, and will restore users from the last backup before the problem.
I have had a look around in the logs and things, it appears there are a few things it could have been - all related to it being php/perl that has been run. There are several scripts in one of the users folders that have hacker-like scripts in them and googling brings up results such as the "Crash_over_ride" backdoor.

How can I stop this (and other backdoors!) in future - I regularly run updates, but its impossible to stop someone uploading something that will try and "attack" the server. I dont want to restrict users too much (ie safemode) as this causes lots of problems. We also have many scripts on the servers that use more powerfull stuff (such as cacti, rddtool and there are many websites running calls to imagemagick and other command line tools) But obously I dont want them to be able to change the root password !!
Click to expand...

Thats not good though. Reinstalling and restoring without knowing how they got in will be your biggest downfall because unless you know how they got in how are you going to stop them from compromising the box again? All they are going to do it re root the box. This is what why i said, find out how they got in first or get somebody to find out for you. Now the damage has been done. By simply doing a restore your back to square one. You have no idea how they got in thus you can expect them to be back. Good luck!

I highly recommend that you get somebody to lock down the box properly so that they cannot root the box again.

Ill send you a PM.
 
We know roughly what caused the problem now - its to do with a bug in the kernal up until not long ago. Due to the kernal files being excluded from the updates we never had the fix.
When we restore we will be leaving off the user(s) which we believe to be the problem for now.

Tom
 
yes - its a bit ironic when it happened really. I've got a list on my desk of things to do to the servers, such as offsite backups, increase security, change root passwords, disable root login from ssh etc.. and mod_security (along with some other bits) is on the list.
We were so busy before christmas that we didnt get round to taking a day out to do these things. I guess that day will be tomorow !
 
Regarding the ssh-login, I changed the ssh config to only accept logins with a private key. This is much safer than the plain password logins. If a user tries to logon without a key, the server will disconnect immediately.

Off course this is only the case if you don't have customers/users which also require SSH login possibilities. Otherwise you'll have to provide them with a private key as well. But in my situation I'm the only one.
 
You must log in or register to reply here.
Back
Top