How to find the perl proces that occupies my cpu

N

nieuwhier

Verified User
Joined
Sep 8, 2005
Messages
280
Location
Netherlands
Hi,

On one of my servers sometimes one or more perl procesess are using up all my CPU capacity.

I don't know what website started this process. The only thing I see is perl process started by apache.

Does anyone has a tip ?

regards,
Michel.
 
1st thing to do: backup all your clients

When there is a proces called perl eating cpu, 99% o/t time your about to be hacked... sorry to tell you that, its not a simple thing.

login as root
--------------------
1st type:
cd /tmp

then type:
ls

do you see ant .txt / .pl files there?

if so: you found the problem, someone hosted a insecuire (most o/t time joomla or mambo) script and someone exploited that so he can use your server as a irc bot or even worse to gain root access and hack it.

i have had it on a server yesterday, the only thing to do about it is move your clients to a tempory server, format your server and let a managing company secuire/hardening your server.(if you dont know how to do that yourself) if thats done you can place your clients back.

If you dont find anything in tmp, post it here and we will look foward into it.
 
Last edited:
I did find two .txt files in the .tmp directory with some .pl code in it.(IRAN HACKERS SABOTAGE Connect Back Shell).

This morning I also disabled a website on this server that was hacked and had the same .txt files in there root directory and a directory with lots of files under it. That link to that directory was included in a spammail.

I checked all sites on that server but did not find anything else so why exactly moving all customers and re-install the server ?
 
Because you never know what the hackers installed else.

The name says it all:
IRAN HACKERS SABOTAGE Connect Back Shell

Connect Back Shell, this means they let your server connect to them so they have shell access.

It is the best to format your server and re-install it, thats the only way to be sure you are 100% clean.
 
Hmm...

I just had alot of processes on a new server of mine, which is hardened by my self. /tmp is protected as well:

[root@vserver6 ~]# /tmp/date
-bash: /tmp/date: Permission denied

i don't think im hacked to be honest, the server is pretty well hardened. Can any one confirm my thoughts? I think its just a user running perl, but who...
 
Well...

Some quick steps
- take a look at the /server-status, usefull info
- Monitor mysql
- check your logs
- run top and check it for a while
- check the process list for strange things
 
Top:

11589 apache 15 0 76224 46m 4568 S 7.3 1.2 0:01.75 httpd
15382 apache 15 0 21804 8348 4556 S 3.7 0.2 0:00.11 php
15380 apache 15 0 21804 8348 4576 S 3.3 0.2 0:00.10 php
15388 apache 15 0 21804 8324 4556 S 3.3 0.2 0:00.10 php
15389 apache 15 0 21804 8188 4412 S 3.3 0.2 0:00.10 php
15391 apache 15 0 21804 8352 4556 S 3.3 0.2 0:00.10 php
15378 apache 15 0 21804 8352 4556 S 2.7 0.2 0:00.08 php
15397 apache 15 0 0 0 0 Z 2.0 0.0 0:00.06 perl <defunct>
15398 apache 15 0 0 0 0 Z 2.0 0.0 0:00.06 perl <defunct>
15404 apache 17 0 0 0 0 Z 2.0 0.0 0:00.06 perl <defunct>
15359 apache 15 0 12036 7232 1276 S 1.7 0.2 0:00.08 perl
15402 apache 18 0 0 0 0 Z 1.7 0.0 0:00.05 perl <defunct>
15358 apache 15 0 12036 7180 1276 S 1.3 0.2 0:00.06 perl
15396 apache 15 0 13068 8184 1212 S 1.3 0.2 0:00.04 perl
15400 apache 15 0 13068 8176 1180 S 1.3 0.2 0:00.04 perl
15405 apache 22 0 0 0 0 Z 1.3 0.0 0:00.04 perl <defunct>
32560 apache 15 0 13068 8772 1816 S 1.3 0.2 0:06.64 perl
15357 apache 15 0 11900 7140 1272 S 1.0 0.2 0:00.06 perl
15362 apache 15 0 12028 7172 1276 S 1.0 0.2 0:00.06 perl
15365 apache 15 0 12044 7184 1276 S 1.0 0.2 0:00.06 perl
15399 apache 15 0 13068 8168 1196 S 1.0 0.2 0:00.03 perl
15401 apache 15 0 13068 8152 1180 S 1.0 0.2 0:00.03 perl
15403 apache 15 0 13068 8100 1128 S 1.0 0.2 0:00.03 perl

but i still dont think im hacked. its like almost impossible. i just got to know who is spawning these perl processes and suspend him :)
 
Try this to see who started it:
pstree -Aup

If its the webserver then do this:
ps ax -o comm,cmd,lstart |grep perl

The youve got the starting time and last is to check the apache logs(or cron) around that time :-)

Good luck!
 
You can't run /tmp/date directly, but the attacker probably launched "perl /tmp/date", which can't be stopped.

Follow the suggestions and clean your system completely: find the vulnerability that started it all (probably an RFI on a PHP application) and fix it, create all backups, do a fresh install, restore the backups.

Next time you have a perl process running and you want to know which script is it running, just run "lsof -p <PID>" as root and look for any REG (regular) file it is reading: one of them is the script.

If you need help with the vulnerability hire a consultant. System security is not a joke; unless you are trained for it, don't expect to have a secure system -- ever.
 
Where can i find and then delete this files ?

[root@server /]# ps ax -o comm,cmd,lstart |grep perl
grep grep perl Tue Nov 3 12:07:00 2009
perl ./ouvxxbr.pl Tue Nov 3 07:30:31 2009
perl ./wmrwrmq.pl Tue Nov 3 10:21:47 2009
perl ./wmrwrmq.pl Tue Nov 3 10:21:47 2009
perl ./wmrwrmq.pl Tue Nov 3 10:21:47 2009
[root@server /]#

[root@server /]# find wmrwrmq.pl
find: wmrwrmq.pl: No such file or directory
[root@server /]# locate wmrwrmq.pl
[root@server /]#

Thanks
 
Using find:
Code: 
find / -name wmrwrmq.pl
Using locate:
Code: 
updatedb
locate wmrmwmq.pl

Some scripts just delete themselves after being launched; you'll have to launch "lsof -p <PID>" and look for the open file descriptor (obviously pointing to a deleted file).
 
Is there anyway to remove/control RFI in website such as wordpress, joomla, etc?
 
Of course there is, but as always more security means more constraints.

You can use an IPS (mod_security2, suhosin, etc) but you'll have to configure it in order to avoid problems with all applications: testing it in monitoring mode (IDS) for some time then removing offending rules and activating the IPS.

You can also disable some PHP dangerous functions via disable_functions but you'll have to check that none of the applications use them. Same thing for blocking functions to load external resources through allow_url_* directives.

You could move from mod_php (if in use) to PHP-CGI to avoid horizontal privilege escalation, but you may have applications that need mod_php (some modules, php_ini directive etc).

You can mount all temporary file systems noexec/nosuid/nodev but some applications, again, need to execute files from those directories.

There are many other ways to secure your system to avoid RFI and code injection, but all of them have constraints and have to be tested intensively before use: that's the reason for which they are not activated by default.
Read all threads on this forum about PHP security to discover them all.
 
I'm having the same problems as "xantini".

There are running a few perl scripts, but they are automatic deleted everytime :( So you can't track them
I'm looking for an option to block "perl" for the /tmp directory.
 
Unfortunately most worms today do not rely on the shebang and executable properties of a script to run themselves, but instead run "/path/to/bin/perl /path/to/script" for which the noexec mount option offers no protection.

There is one way to protect perl from being runned by untrusted users: chmod it o-x (no execution rights to others), create a dedicated group (runperl or something), chown it to that group and add every trusted user that uses perl (there may be many; monitor your system) to it. For example the user running exim, which uses /etc/exim.pl in DA.

Most administrators just run a looped script that checks via "ps" if a new and unknown perl script is being runned, kill it and send an Email to them (see CSF and many others).
 
I think this command will help determine where the script is running:

Code: 
ps ax -fH eww | grep perl
 
That will show PWD, but like any other environment variable it can be modified by the script. It's worth a try anyway, but it's better (faster and clearer) to rely on files within /proc/<PID>/ than to play with ps's switches.
 
You must log in or register to reply here.
Back
Top