[Solved] Rspamd Fails on CloudLinux 9: Solution for "cannot allocate scratch space for hyperscan" Crash & CustomBuild Fix Suggestion

[souzadavi]

Hello DirectAdmin Community,

I am writing to document a critical issue we recently faced with Rspamd failing to start on a CloudLinux 9 server, managed by DirectAdmin. We have successfully resolved it with the help of CloudLinux support, and I am sharing the details here to help others and to propose a potential improvement for the CustomBuild script.

1. The Problem​

After installing Rspamd using da build rspamd on a production server with CloudLinux 9, the service would immediately crash upon starting.

• The systemctl status rspamd command showed the service failing with Result: core-dump and signal=ABRT.
• The Rspamd log (/var/log/rspamd/rspamd.log) consistently displayed the following fatal error:
  
  rspamd_multipattern_compile: cannot allocate scratch space for hyperscan: error code -5<br>rspamd_url_init: cannot compile url matcher static patterns, fatal error: cannot allocate scratch space for hyperscan: error code -5<br>

We performed extensive troubleshooting, including full system updates (dnf update), clean reinstalls of Rspamd and its dependencies (including hyperscan), and verifying server resources (62GB RAM) and CPU compatibility (avx2, sse4_2), but the issue persisted.

2. The Root Cause: Shared Library Conflict​

The investigation by CloudLinux support identified the root cause as a shared library conflict. The Rspamd process was dynamically linking against an older, incompatible version of the Hyperscan library provided by CloudLinux's alternate packages, instead of the newer system version.

• Incorrect Library Being Loaded: alt-hyperscanpd-5.4.0-8.el9.x86_64 (likely installed as a dependency for an alt-php package).
• Correct System Library Required: hyperscan-5.4.1-2.el9.x86_64.

The system's library loader was prioritizing the path containing the alt- version, causing the Rspamd service to crash.

3. The Solution​

The fix involves forcing the Rspamd systemd service to prioritize the correct system library path (/usr/lib64) by setting the LD_LIBRARY_PATH environment variable. This is achieved by creating a systemd override file.

Step-by-step fix:

1. Create the override directory for the Rspamd service:
   
   Bash
   mkdir -p /etc/systemd/system/rspamd.service.d/<br>
2. Create a configuration file inside this directory:
   
   Bash
   touch /etc/systemd/system/rspamd.service.d/override.conf<br>
3. Add the following content to override.conf:
   
   Ini, TOML
   <span>[Service]</span><br><span>Environment</span>=<span>"LD_LIBRARY_PATH=/usr/lib64"</span><br>
4. Reload the systemd configuration and restart Rspamd:
   
   Bash
   systemctl daemon-reload<br>systemctl restart rspamd<br>

After applying this override, Rspamd started successfully and has been running stable.

4. Suggestion for CustomBuild Enhancement​

To prevent other DirectAdmin users on CloudLinux from facing this hard-to-diagnose issue, I would like to suggest an enhancement to the da build rspamd script. The script could be made more robust by automatically detecting this specific environment and applying the fix proactively. The logic could be as follows:

1. Check if the OS is CloudLinux (e.g., [ -f /etc/cloudlinux-release ]).
2. Check if an alt-hyperscan package is installed (e.g., rpm -q alt-hyperscanpd).
3. If both conditions are true, the da build rspamd script could automatically create the /etc/systemd/system/rspamd.service.d/override.conf file with the necessary LD_LIBRARY_PATH directive.

This would make the Rspamd installation on CloudLinux environments seamless and prevent future support tickets for the same issue.

Hope this detailed report helps the community and the DirectAdmin team.

Thank you!