Preventing malicious traffic from bc.googleusercontent.com

After my last post I wrote a cron script to retrieve and block googleusercontent ip's with CSF/LFD.

Add in /etc/csf/csf.blocklists (or use the CSF web ui)
Code: 
GOOGLECLOUD|86400|0|https://domain.com/blocklists/bcgoogle-cloud_ips.txt
Use a domain you own or have access to. (If the blocklist isn't loaded afterwards by CSF, then check folder and file permissions)

create /etc/cron.daily/parse_bcgoogle-cloud.py and make it executable.

Contents of file:
Code: 
#!/usr/bin/env python3

import json
import urllib.request
import subprocess
from pathlib import Path

URL = "https://www.gstatic.com/ipranges/cloud.json"

OUTPUT_PATH = Path(
    "/home/admin/domains/domain.com/public_html/blocklists/bcgoogle-cloud_ips.txt"
)

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0.0.0 Safari/537.36"
    )
}

try:
    req = urllib.request.Request(URL, headers=HEADERS)

    with urllib.request.urlopen(req, timeout=30) as response:
        data = json.loads(response.read().decode("utf-8"))

    ip_list = []

    for item in data.get("prefixes", []):
        ip = item.get("ipv4Prefix") or item.get("ipv6Prefix")
        if ip:
            ip_list.append(ip)

    # Remove duplicates and sort for consistent output
    ip_list = sorted(set(ip_list))

    new_content = "\n".join(ip_list) + "\n"

    # Read existing content if the file exists
    if OUTPUT_PATH.exists():
        old_content = OUTPUT_PATH.read_text()
    else:
        old_content = ""

    # Only update when something changed
    if new_content != old_content:

        OUTPUT_PATH.write_text(new_content)

        print(
            f"Updated {len(ip_list)} unique Google-cloud IPs/CIDRs "
            f"in {OUTPUT_PATH}"
        )

    else:
        print("No changes detected.")

except subprocess.CalledProcessError as e:
    print(f"Failed to restart CSF/LFD: {e}")

except Exception as e:
    print(f"Error processing IP list: {e}")

Originally I restarted CSF with this script, but realized this was not necessary, because CSF will read the blocklist automatically according to the settings in
Code: 
GOOGLECLOUD|86400|0|https://domain.com/blocklists/bcgoogle-cloud_ips.txt

So this is leftover code (I left it as is, because I have to test if the script still works without) :
Code: 
except subprocess.CalledProcessError as e:
    print(f"Failed to restart CSF/LFD: {e}")

But as lazy as I am, I thought hey my script is working so why spend energy in optimizing as I am not a very good coder. ;) And yes I had help from AI.

After setting this up execute ./parse_bcgoogle-cloud.py and check if https://domain.com/blocklists/bcgoogle-cloud_ips.txt is created, if not check file and folder permissions and execute again.

Then execute csf -ra and check in /var/lib/csf if csf.block.GOOGLECLOUD is created

If you have improvements or suggestions, you're welcome to share.
 
I have PHP script for you guy without needed the cronjob, also this is AI Code, the script will have self-cache to ".bcgoogle-cloud-cache" directory

Code: 
<?php
declare(strict_types=1);

/**
 * Google Cloud -> CSF/LFD blocklist. Requires PHP 8.0+ and ext-curl.
 * Upload as /blocklists/bcgoogle-cloud_ips.php.
 * PHP must be able to create/write the cache directory beside this file.
 * A missing cache-directory .htaccess is automatically created on every request
 * with "Require all denied". Apache 2.4 must allow AuthConfig overrides there.
 * Existing .htaccess files are preserved.
 * Both CIDR ranges and plain IPv4/IPv6 values are accepted and emitted as given.
 *
 * /etc/csf/csf.blocklists:
 * GOOGLECLOUD|3600|0|https://domain.com/blocklists/bcgoogle-cloud_ips.php
 *
 * The PHP cache lasts 3600 seconds. Refresh happens on the next request after
 * expiry; no cron is required. The CSF example also uses a 3600-second interval.
 * Failed refreshes preserve and serve the last valid cache, even after expiry.
 * With no valid cache available, failures return HTTP 503.
 */

ini_set('display_errors', '0');
header('Content-Type: text/plain; charset=UTF-8');
header('X-Content-Type-Options: nosniff');
header('Cache-Control: no-store');

$sourceUrl = 'https://www.gstatic.com/ipranges/cloud.json';
$cacheTtl = 3600;
$cacheDirectory = __DIR__ . '/.bcgoogle-cloud-cache';
$cacheFile = $cacheDirectory . '/google-cloud.txt';

function googleCloudPrepareCacheDirectory(string $directory): void
{
    if (!is_dir($directory) && !@mkdir($directory, 0700, true) && !is_dir($directory)) {
        throw new RuntimeException('Cannot create cache directory: ' . $directory);
    }

    $htaccess = $directory . '/.htaccess';
    if (is_file($htaccess)) {
        return;
    }

    // Exclusive creation preserves a file another request may have just created.
    $handle = @fopen($htaccess, 'x');
    if ($handle === false) {
        clearstatcache(true, $htaccess);
        if (is_file($htaccess)) {
            return;
        }
        throw new RuntimeException('Cannot create cache-directory .htaccess.');
    }

    $complete = false;
    try {
        $rules = "Require all denied\n";
        if (@fwrite($handle, $rules) !== strlen($rules) || !@fflush($handle)) {
            throw new RuntimeException('Cannot write cache-directory .htaccess.');
        }
        $complete = true;
    } finally {
        fclose($handle);
        if (!$complete) {
            @unlink($htaccess);
        }
    }
}

function googleCloudValidIpOrCidr(string $address, ?int $family = null): bool
{
    $parts = explode('/', $address);
    if (count($parts) > 2) {
        return false;
    }

    $family = $family ?? (strpos($parts[0], ':') === false ? 4 : 6);
    $flag = $family === 4 ? FILTER_FLAG_IPV4 : FILTER_FLAG_IPV6;
    if (filter_var($parts[0], FILTER_VALIDATE_IP, $flag) === false) {
        return false;
    }

    // A plain IP remains a plain IP in the generated blocklist.
    if (count($parts) === 1) {
        return true;
    }

    // A default route (/0) is deliberately rejected for this blocklist.
    if (preg_match('/\A[1-9][0-9]{0,2}\z/', $parts[1]) !== 1) {
        return false;
    }

    return (int) $parts[1] <= ($family === 4 ? 32 : 128);
}

function googleCloudReadCache(string $path): ?array
{
    $handle = @fopen($path, 'rb');
    if ($handle === false) {
        return null;
    }

    try {
        $stat = fstat($handle);
        $body = stream_get_contents($handle);
    } finally {
        fclose($handle);
    }

    if ($stat === false || $body === false || $body === '') {
        return null;
    }

    foreach (explode("\n", rtrim($body, "\r\n")) as $line) {
        if (!googleCloudValidIpOrCidr(rtrim($line, "\r"))) {
            return null;
        }
    }

    return ['body' => $body, 'time' => (int) $stat['mtime']];
}

function googleCloudDownload(string $url): string
{
    if (!extension_loaded('curl')) {
        throw new RuntimeException('The PHP cURL extension is required.');
    }

    $curl = curl_init($url);
    if ($curl === false) {
        throw new RuntimeException('Could not initialize cURL.');
    }

    $body = '';
    try {
        curl_setopt_array($curl, [
            CURLOPT_CONNECTTIMEOUT => 5,
            CURLOPT_TIMEOUT => 20,
            CURLOPT_FOLLOWLOCATION => false,
            CURLOPT_SSL_VERIFYPEER => true,
            CURLOPT_SSL_VERIFYHOST => 2,
            CURLOPT_HTTPHEADER => ['Accept: application/json'],
            CURLOPT_USERAGENT => 'GoogleCloud-CSF-Blocklist/1.0',
            CURLOPT_WRITEFUNCTION => static function ($handle, string $chunk) use (&$body): int {
                $body .= $chunk;
                return strlen($chunk);
            },
        ]);

        if (curl_exec($curl) === false) {
            throw new RuntimeException('Google feed request failed: ' . curl_error($curl));
        }
        $status = (int) curl_getinfo($curl, CURLINFO_HTTP_CODE);
        if ($status !== 200) {
            throw new RuntimeException('Google feed returned HTTP ' . $status . '.');
        }
    } finally {
        unset($curl);
    }

    $data = json_decode($body, true, 512, JSON_THROW_ON_ERROR);
    if (!is_array($data) || !is_array($data['prefixes'] ?? null) || $data['prefixes'] === []) {
        throw new RuntimeException('Google feed has no valid prefixes array.');
    }

    $addresses = [];
    foreach ($data['prefixes'] as $entry) {
        if (!is_array($entry)) {
            throw new RuntimeException('Google feed contains an invalid prefix entry.');
        }
        $found = false;
        foreach (['ipv4Prefix' => 4, 'ipv6Prefix' => 6] as $key => $family) {
            if (!array_key_exists($key, $entry)) {
                continue;
            }
            $address = $entry[$key];
            if (!is_string($address) || !googleCloudValidIpOrCidr($address, $family)) {
                throw new RuntimeException('Google feed contains an invalid IP address or CIDR.');
            }
            $addresses[$address] = true;
            $found = true;
        }
        if (!$found) {
            throw new RuntimeException('Google feed contains an entry without an IP address or CIDR.');
        }
    }

    $lines = array_keys($addresses);
    sort($lines, SORT_STRING);
    return implode("\n", $lines) . "\n";
}

function googleCloudWriteCache(string $directory, string $path, string $body): void
{
    $temp = @tempnam($directory, 'feed-');
    if ($temp === false) {
        throw new RuntimeException('Cannot create a temporary cache file.');
    }
    try {
        $expectedDirectory = realpath($directory);
        if ($expectedDirectory === false || realpath(dirname($temp)) !== $expectedDirectory) {
            throw new RuntimeException('Temporary cache file is outside the cache directory.');
        }
        if (@file_put_contents($temp, $body) !== strlen($body)) {
            throw new RuntimeException('Cannot write the complete cache file.');
        }
        if (!@rename($temp, $path)) {
            throw new RuntimeException('Cannot replace the cache file.');
        }
    } finally {
        if (is_file($temp)) {
            @unlink($temp);
        }
    }
}

// Protect the directory before both fresh-cache responses and refreshes.
try {
    googleCloudPrepareCacheDirectory($cacheDirectory);
} catch (Throwable $error) {
    error_log('[GoogleCloud blocklist] ' . $error->getMessage());
    http_response_code(503);
    header('Retry-After: 60');
    header('X-Blocklist-Cache: UNAVAILABLE');
    echo "# Google Cloud cache protection is unavailable.\n";
    exit;
}

$cache = googleCloudReadCache($cacheFile);
$cacheStatus = 'HIT';
$lock = null;
$locked = false;

if ($cache === null || time() - $cache['time'] >= $cacheTtl) {
    try {
        $lock = @fopen($cacheDirectory . '/refresh.lock', 'c');
        if ($lock === false) {
            throw new RuntimeException('Cannot open the cache refresh lock.');
        }

        // Cold requests wait for the first refresh; other requests can serve stale data.
        $locked = flock($lock, $cache === null ? LOCK_EX : LOCK_EX | LOCK_NB);
        if ($locked) {
            // Another request may have refreshed while this request was waiting.
            $cache = googleCloudReadCache($cacheFile) ?? $cache;
            if ($cache === null || time() - $cache['time'] >= $cacheTtl) {
                $body = googleCloudDownload($sourceUrl);
                googleCloudWriteCache($cacheDirectory, $cacheFile, $body);
                $cache = ['body' => $body, 'time' => time()];
                $cacheStatus = 'REFRESH';
            }
        } elseif ($cache !== null) {
            $cacheStatus = 'STALE';
        } else {
            throw new RuntimeException('Cannot acquire the cache refresh lock.');
        }
    } catch (Throwable $error) {
        error_log('[GoogleCloud blocklist] ' . $error->getMessage());
        $cacheStatus = 'STALE';
    } finally {
        if (is_resource($lock)) {
            if ($locked) {
                flock($lock, LOCK_UN);
            }
            fclose($lock);
        }
    }
}

if ($cache === null) {
    http_response_code(503);
    header('Retry-After: 60');
    header('X-Blocklist-Cache: UNAVAILABLE');
    echo "# Google Cloud blocklist temporarily unavailable.\n";
    exit;
}

header('X-Blocklist-Cache: ' . $cacheStatus);
header('X-Blocklist-Cache-Age: ' . max(0, time() - $cache['time']));
echo $cache['body'];
 
Anecdotally, GCP abuse seems to have died down a bit on my side. I used to see 10+ notifications from CSF a day from the GCP network and I don't recall the last one I saw.

Did they finally get their house in order?
 
You must log in or register to reply here.
Back
Top