Script to detect illegal apache processes

floyd

Verified User
Joined
Mar 29, 2005
Messages
6,248
Edit: I am back to maintaining this script again since the one at webhostgear is dead now. It cannot even be downloaded anymore.


Here is a perl script that will detect illegal or unknown apache processes. It
will kill them and send you an email letting you know it killed them and what
they were. It gives you the line from "ps aux" and the information from
"ls -l /proc/$pid" This should help you in quickly identifying problems.

Comments are welcome good or bad. It would be especially useful if anybody
has any ideas on how to improve the script. I know it has been very useful
to me. Because of it I detected the recent roundcube exploit very quickly
and was relatively unaffected by it. Not all of my servers were running this so
I was affected but not nearly as bad as it could have been.

Anyway here is the script:

EDIT I am just going to start giving a date as to when this script is updated.
The date will be in the script itself that way you always know which one you have.

Code:
#!/usr/bin/perl

# Last updated January, 26 2009 1232972865

# This script will check the processes running as apache every so many 
# seconds
# It has been trested on RedHat based systems.
# Create a file and copy and paste these contents and put it wherever you 
# want it
# chmod 755 the script
# So that the script will start on boot add to /etc/rc.d/rc.local the command 
# "/path/to/script/ &"
# Start the script from the command line use "/path/to/script/ &"
# To stop the script "ps aux | grep apache.pl" and then kill the process id.
# I will not be held responsible for use of this script.
# Everybody who wants to use it should read through it so they know what 
# its doing.

# Configure variables

# The ip address of the server. This 
# identifies which of your servers has a problem
$ip = "xxx.xxx.xxx.xxx"; 

# Your email. Make sure you have a \ before the @
# Make sure mailx is installed.
$email = "email\@example.com"; 

# Subject of the email sent to you
$subject = "script running";

# $test on will email you all the processes currently running 
# as apache and check to make sure you will get the email
$test = "off"; # or "on"

# You want it to kill the process
$kill = "yes"; # or "no" If "no" it will just notify you the script is running

# Checks every so many seconds
$time = 10;

# Include processes you want to be ignored. One per line between the (   );
# The are some others that you might want to put here.
# When you get the email you can decide at that point if the process needs 
# to be added here.
@exclude = qw(
defunct
/usr/sbin/httpd
/usr/bin/sendmail
/usr/sbin/sendmail
);



foreach $exclude(@exclude){
        $exclude2 .= " | grep -v \"$exclude\"";
}


# Edit anything after here at your own risk

chomp(@exclude);

# Test if OS is FreeBSD
if (`uname -s` eq "FreeBSD") {
        print "Sorry this program is not compatible with FreeBSD!\n";
        exit;
}

while(){

$bodyofemail = "";


if ($test eq "off"){

        @grep = `ps aux | grep ^apache $exclude2`;

}

if ($test eq "on"){
        @grep = `ps aux | grep ^apache`;
}


chomp(@grep);

foreach $grep(@grep){

        ($user,$pid) = split(/\s+/, $grep);

        if ($grep ne ""){

                print "$grep\n";

                @ls = `ls -l /proc/$pid`;
                $bodyofemail .= "$grep\n\n@ls\n\n\n";
        }


        if (($test eq "off")&&($kill eq "yes")){

                `kill -9 $pid`;

        }

}

if($bodyofemail){

        `echo "$date\n\n$ip\n\n$bodyofemail" | mail -s "$subject" $email`;

}

sleep($time);
}
 
Last edited:
How about moving it to the How-To section? Would that work for everyone?

Jeff
 
Floyd you forgot to put a ";" after the subject line :P

Code:
root@http5:~/bin# ./check-apache.pl 
Scalar found where operator expected at ./check-apache.pl line 31, near "$test"
        (Missing semicolon on previous line?)
syntax error at ./check-apache.pl line 31, near "$test "
Execution of ./check-apache.pl aborted due to compilation errors.
 
That's what happens when you have an idea while you are making the post and you don't test it first.
 
Ok this code seems to work for testing if the OS is freebsd or not:

Code:
$uname = `uname -s`;
if ($uname =~ /FreeBSD/i) {
        print "Sorry this program is not compatible with FreeBSD!\n";
        exit;
}
 
I have a question what is the purpose of looking at /proc/$PID

This is the info I get:

Code:
www-data 22944  0.0  0.8  48800 16976 ?        S    15:47   0:00 /usr/sbin/apache2 -k start

total 0
 dr-xr-xr-x 2 www-data www-data 0 Jan 14 16:02 attr
 -r-------- 1 root     root     0 Jan 14 16:02 auxv
 -r--r--r-- 1 root     root     0 Jan 14 16:02 cgroup
 --w------- 1 root     root     0 Jan 14 16:02 clear_refs
 -r--r--r-- 1 root     root     0 Jan 14 15:47 cmdline
 -rw-r--r-- 1 root     root     0 Jan 14 16:02 coredump_filter
 -r--r--r-- 1 root     root     0 Jan 14 16:02 cpuset
 lrwxrwxrwx 1 root     root     0 Jan 14 16:02 cwd -> /
 -r-------- 1 root     root     0 Jan 14 16:02 environ
 lrwxrwxrwx 1 root     root     0 Jan 14 16:02 exe -> /usr/sbin/apache2
 dr-x------ 2 root     root     0 Jan 14 16:02 fd
 dr-x------ 2 root     root     0 Jan 14 16:02 fdinfo
 -r--r--r-- 1 root     root     0 Jan 14 16:02 io
 -r-------- 1 root     root     0 Jan 14 16:02 limits
 -rw-r--r-- 1 root     root     0 Jan 14 16:02 loginuid
 -r--r--r-- 1 root     root     0 Jan 14 16:02 maps
 -rw------- 1 root     root     0 Jan 14 16:02 mem
 -r--r--r-- 1 root     root     0 Jan 14 16:02 mounts
 -r-------- 1 root     root     0 Jan 14 16:02 mountstats
 -rw-r--r-- 1 root     root     0 Jan 14 16:02 oom_adj
 -r--r--r-- 1 root     root     0 Jan 14 16:02 oom_score
 lrwxrwxrwx 1 root     root     0 Jan 14 16:02 root -> /
 -rw-r--r-- 1 root     root     0 Jan 14 16:02 sched
 -r--r--r-- 1 root     root     0 Jan 14 16:02 smaps
 -r--r--r-- 1 root     root     0 Jan 14 15:47 stat
 -r--r--r-- 1 root     root     0 Jan 14 16:02 statm
 -r--r--r-- 1 root     root     0 Jan 14 15:47 status
 dr-xr-xr-x 3 www-data www-data 0 Jan 14 16:02 task
 -r--r--r-- 1 root     root     0 Jan 14 16:02 wchan

What is this information it gathered from /proc/$PID supposed to tell me?
 
I don't know what all of it means but if you look at my first post here you can see that it gave me the clue that something was wrong with roundcube. Many time you can get a clue as to where the real problem is.

It just gives you extra information about the process before it kills it.

I am sure somebody can add to the explanation.

Here are some examples:

Code:
lrwxrwxrwx 1 apache apache 0 Jan 14 15:35 cwd -> /tmp/  /mech-linux
lrwxrwxrwx 1 apache apache 0 Jan 14 15:35 exe -> /tmp/  /mech-linux/ps-x
lrwxrwxrwx 1 apache apache 0 Jan 14 15:35 exe -> /tmp/cback
lrwxrwxrwx 1 apache apache 0 Jan 14 15:44 cwd -> /var/tmp/.,./.bash
lrwxrwxrwx 1 apache apache 0 Jan 14 15:44 exe -> /var/tmp/.,./.bash/ntpd
lrwxrwxrwx 1 apache apache 0 Jan 14 15:01 exe -> /var/tmp/.,./bind/19876
 
Also if "/usr/sbin/apache2" is legit you want to add that to the exclude array.
 
If anybody knows how to make this compatible with FreeBSD or any other OS please let me know.
 
hello,

I'm trying to get your script to work on Debian (because I WAS affected by the RoundCube bug :mad:).

When i set test to on (to get the first test email) it says "sh: line 274: mail: command not found".

What should i install to get this to work?

thx,
Stijn
 
Looks like it ends up in a loop when having 'kill' disabled.

Every email adds another entry about the same pid, till it get out of memory.

apache 9576 0.0 0.0 2304 968 pts/1 S+ 14:25 0:00 vi test

total 0
-r-------- 1 apache apache 0 Jan 25 14:25 auxv
-r--r--r-- 1 apache apache 0 Jan 25 14:25 cmdline
-r--r--r-- 1 apache apache 0 Jan 25 14:25 cpuset
lrwxrwxrwx 1 apache apache 0 Jan 25 14:25 cwd -> /test/scripts
-r-------- 1 apache apache 0 Jan 25 14:25 environ
lrwxrwxrwx 1 apache apache 0 Jan 25 14:25 exe -> /bin/vi
dr-x------ 2 apache apache 0 Jan 25 14:25 fd
?r--r--r-- 1 apache apache 0 Jan 25 14:25 io
-rw-r--r-- 1 apache apache 0 Jan 25 14:25 loginuid
-r--r--r-- 1 apache apache 0 Jan 25 14:25 maps
-rw------- 1 apache apache 0 Jan 25 14:25 mem
-r--r--r-- 1 apache apache 0 Jan 25 14:25 mounts
-r-------- 1 apache apache 0 Jan 25 14:25 mountstats
-r--r--r-- 1 apache apache 0 Jan 25 14:25 numa_maps
-rw-r--r-- 1 apache apache 0 Jan 25 14:25 oom_adj
-r--r--r-- 1 apache apache 0 Jan 25 14:25 oom_score
lrwxrwxrwx 1 apache apache 0 Jan 25 14:25 root -> /
-r--r--r-- 1 apache apache 0 Jan 25 14:25 schedstat
-r-------- 1 apache apache 0 Jan 25 14:25 smaps
-r--r--r-- 1 apache apache 0 Jan 25 14:25 stat
-r--r--r-- 1 apache apache 0 Jan 25 14:25 statm
-r--r--r-- 1 apache apache 0 Jan 25 14:25 status
dr-xr-xr-x 3 apache apache 0 Jan 25 14:25 task
-r--r--r-- 1 apache apache 0 Jan 25 14:25 wchan



apache 9576 0.0 0.0 2304 968 pts/1 S+ 14:25 0:00 vi test

total 0
-r-------- 1 apache apache 0 Jan 25 14:25 auxv
-r--r--r-- 1 apache apache 0 Jan 25 14:25 cmdline
-r--r--r-- 1 apache apache 0 Jan 25 14:25 cpuset
lrwxrwxrwx 1 apache apache 0 Jan 25 14:25 cwd -> /test/scripts
-r-------- 1 apache apache 0 Jan 25 14:25 environ
lrwxrwxrwx 1 apache apache 0 Jan 25 14:25 exe -> /bin/vi
dr-x------ 2 apache apache 0 Jan 25 14:25 fd
?r--r--r-- 1 apache apache 0 Jan 25 14:25 io
-rw-r--r-- 1 apache apache 0 Jan 25 14:25 loginuid
-r--r--r-- 1 apache apache 0 Jan 25 14:25 maps
-rw------- 1 apache apache 0 Jan 25 14:25 mem
-r--r--r-- 1 apache apache 0 Jan 25 14:25 mounts
-r-------- 1 apache apache 0 Jan 25 14:25 mountstats
-r--r--r-- 1 apache apache 0 Jan 25 14:25 numa_maps
-rw-r--r-- 1 apache apache 0 Jan 25 14:25 oom_adj
-r--r--r-- 1 apache apache 0 Jan 25 14:25 oom_score
lrwxrwxrwx 1 apache apache 0 Jan 25 14:25 root -> /
-r--r--r-- 1 apache apache 0 Jan 25 14:25 schedstat
-r-------- 1 apache apache 0 Jan 25 14:25 smaps
-r--r--r-- 1 apache apache 0 Jan 25 14:25 stat
-r--r--r-- 1 apache apache 0 Jan 25 14:25 statm
-r--r--r-- 1 apache apache 0 Jan 25 14:25 status
dr-xr-xr-x 3 apache apache 0 Jan 25 14:25 task
-r--r--r-- 1 apache apache 0 Jan 25 14:25 wchan
 
Every email adds another entry about the same pid, till it get out of memory.

I am not sure what you mean by "adds another entry." If kill is disabled it should just sent you an email every x number of seconds. The script does not add any extra load if kill is disabled unless the mail cannot be delivered. Did your server load increase?
 
What I meant to say is..
If I test the script by disabling 'killing' in the script and start a process as apache:

sudo -u apache less /etc/passwd

I get a mail from the script warning me about this process.
Then I get second mail , mentioning the same process twice.. (same pid)
Then I get a third mail, mentioning the same process three times.. etc (same pid)

Even after stopping the test 'less' process, I keep on getting mails with the same growing trend of repetitions as mentioned above.

So il looks like @grep never gets emptied.
 
I tried running the script as is and running the command

sudo -u apache less /etc/passwd

It kept sending me email until I killed the apache owned process. When I killed it the email stopped.

The script also prints the ps line containing the apache owned process and again when I killed it the printing stopped.

You could put

@grep = ();

at the beginning of the while statement to make sure @grep is empty with each loop. Try that and see what happens.

If you still get the email or it prints the same process I am inclined to think that the apache owned process is still running.

But regardless it should not eat up the memory unless more stuff is being added to @grep with each loop.

I can't recreate it. Maybe somebody else can and contribute a fix. I have only tested it on RedHat based machines.
 
I tried running the script as is and running the command

sudo -u apache less /etc/passwd

It kept sending me email until I killed the apache owned process. When I killed it the email stopped.

- correction: it kept on mailing me, the repetition in the mail did not increase anymore.

The script also prints the ps line containing the apache owned process and again when I killed it the printing stopped.

You could put

@grep = ();

- Did not help.

at the beginning of the while statement to make sure @grep is empty with each loop. Try that and see what happens.

If you still get the email or it prints the same process I am inclined to think that the apache owned process is still running.

- I think I found it. it was $bodyofemail which never got emptied.. it kept on appending.

this fixed it for me:

while(){

$bodyofemail = "";
 
after running the script for a while on the foreground, I started getting:

sh: fork: Cannot allocate memory

tracking what could cause this, I found grep's being executed which kept on repeating the "-v" arguments. growing with each execution.

So I added:

$exclude2 = "";

after the while(){
 
after running the script for a while on the foreground, I started getting:

sh: fork: Cannot allocate memory

tracking what could cause this, I found grep's being executed which kept on repeating the "-v" arguments. growing with each execution.

So I added:

$exclude2 = "";

after the while(){

That means nothing will be excluded and kill all your legitimate apache processes. Script edited above.
 
Back
Top