PHP-FPM I/O Monitor Script for DirectAdmin

[kalmaarikalmer]

Hi, asking for recommendations or improvements to monitor and log user php processes in a shared hosting. This script was made to dump overloading clients with info and possibly have their web developers take a look and fix their issues.

Summary:

This script monitors PHP-FPM processes of a specified user, identifying those with high I/O usage. Once a process surpasses the set I/O threshold, the script employs strace to trace its system calls, capturing the results in a log file. Designed especially for administrators and developers engaged with DirectAdmin, this script offers valuable insights into processes consuming a significant amount of I/O resources.

Features:

• User-Specific PHP-FPM Monitoring: Tailored monitoring of PHP-FPM processes for a particular user.
• Child Process Retrieval: Automatically fetches child processes linked to a PHP-FPM process in a recursive manner.
• Configurable I/O Threshold: Set a custom I/O usage percentage that, when exceeded, activates strace.
• Resource Management: Imposes a limit on concurrently running strace processes to maintain system performance.
• Customizable Monitoring Intervals: The interval between I/O checks can be adjusted according to user preference.
• Robust Error Handling: Incorporated checks ensure required utilities are installed, and specified users exist.
• Cleanup Mechanism: The script periodically scans for and terminates strace processes that are no longer needed, optimizing resource utilization.

Usage:

Run the script with the following options:

• --user : Denotes the user whose PHP-FPM processes are to be monitored. (Required)
• --logpath : Specifies the directory where the tracing logs will be saved. This is a mandatory parameter and must always be provided.
• --iothreshold : Establish a custom I/O usage threshold (in percentage) that initiates strace. By default, it's set at 50%.
• --sleep : Dictate the sleep interval in seconds between consecutive checks. It's set to 60 seconds if not defined.
• --help: Showcases the help message detailing available options.
• --maxstrace: Limit the number of concurrent strace processes.

Example Usage:

To scrutinize the user 'john', save logs in '/var/logs', set an I/O threshold of 70%, and execute a check every 45 seconds, use:


./script.sh --user john --logpath /var/logs --iothreshold 70 --sleep 45

Dependencies:

To ensure the smooth operation of the script, verify the installation of both iotop and strace utilities on the system.

Note:

For optimal performance, the script might need elevated privileges, especially when accessing tools such as iotop. Make certain to possess the requisite permissions or contemplate running the script via sudo.

#!/bin/bash

# Default values
LOG_DIR=""
USERNAME=""
SHOW_HELP=false
IO_THRESHOLD=50
SLEEP_INTERVAL=60
MAX_STRACE=10

# Check for necessary utilities
if ! command -v strace &> /dev/null || ! command -v iotop &> /dev/null; then
    echo "Missing required utilities. Ensure both strace and iotop are installed."
    exit 1
fi

# Parse command line arguments
while [[ $# -gt 0 ]]; do
    key="$1"
    case $key in
        --user)
        USERNAME="$2"
        shift
        shift
        ;;
        --logpath)
        LOG_DIR="$2"
        shift
        shift
        ;;
        --iothreshold)
        IO_THRESHOLD="$2"
        shift
        shift
        ;;
        --sleep)
        SLEEP_INTERVAL="$2"
        shift
        shift
        ;;
        --maxstrace)
        MAX_STRACE="$2"
        shift
        shift
        ;;
        --help)
        SHOW_HELP=true
        shift
        ;;
        *)
        echo "Invalid argument: $1"
        echo "Use --help to see available commands."
        exit 1
        ;;
    esac
done

# Show help and exit
if $SHOW_HELP || [ -z "$USERNAME" ] || [ -z "$LOG_DIR" ]; then
    echo "Usage: ./script.sh --user  --logpath  [--iothreshold ] [--sleep ] [--maxstrace ]"
    echo "  --user       : Specify the username for which to trace php-fpm processes. (Required)"
    echo "  --logpath    : Specify the directory where logs should be stored. (Required)"
    echo "  --iothreshold: Define the I/O usage percentage that triggers strace. Defaults to 50%."
    echo "  --sleep      : Define the sleep interval in seconds between checks. Defaults to 60 seconds."
    echo "  --maxstrace  : Define the maximum number of concurrent strace processes. Defaults to 5."
    echo "  --help       : Display this help message."
    exit 0
fi

# Create log directory if it doesn't exist
mkdir -p $LOG_DIR

get_children() {
    local parent=$1
    for child in $(pgrep -P $parent); do
        echo $child
        get_children $child
    done
}

while true; do
    # Print a timestamped message for each refresh
    echo "Checking for high I/O php-fpm processes and their children... (Timestamp: $(date))"

    # Get all php-fpm PIDs for specified user
    PHP_FPM_PIDS=$(pgrep -u $USERNAME php-fpm)

    # Accumulate all child PIDs of these php-fpm processes
    ALL_PIDS=""
    for PID in $PHP_FPM_PIDS; do
        ALL_PIDS+="$PID "
        ALL_PIDS+=$(get_children $PID)
        ALL_PIDS+=" "
    done

    # Check I/O for each PID and limit concurrent strace processes
    CURRENT_STRACE_COUNT=0
    for PID in $ALL_PIDS; do
        # Check if the process is using over the specified I/O threshold
        IO_USAGE=$(sudo iotop -b -n 1 -P | awk -v pid=$PID '$1 == pid && $10 > '$IO_THRESHOLD' {print $10}')
        if [[ ! -z $IO_USAGE ]] && [[ $CURRENT_STRACE_COUNT -lt $MAX_STRACE ]]; then
            LOG_FILE="$LOG_DIR/${USERNAME}.log.$PID"
           
            # Trace the process
            echo "Tracing high I/O PID $PID..."
            strace -p $PID -o $LOG_FILE &
            ((CURRENT_STRACE_COUNT++))
        fi
    done

    # Sleep for a while before checking again
    sleep $SLEEP_INTERVAL
done