high load with modsecurity

A

apogee

Verified User
Joined
Jul 6, 2019
Messages
193
Location
EU
Since nov. 19 we have a crazy high load when modsec is enabled with the comodo rulset:
Code: 
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
8758 apache    20   0 5971712 170564   2840 S 599.7  0.3 910:20.11 httpd
8682 apache    20   0 5775076 155780   2440 S  99.7  0.2 119:30.49 httpd

The machine seems to write abnormally much on the storage on /tmp, 6 TB in one week:

Code: 
[root@da-dev1 ~]# ls -laht /tmp/

-rw-r-----   1 apache  apache  515P Jan  7 15:05 apache-ip.pag
-rw-r-----   1 apache  apache  257T Jan  7 13:10 apache-ip.dir


DA support says this is my problem because they didn't develop the rulset.

I only use the server for tests with 2 active accounts, the server is almost inactive but as soon as a static html page is loaded it blows up my CPUs and it is writes constantly in /tmp.

I somehow can't imagine that I am the only one with this problem, do any of you have such problems?

Currently I have simply disabled the module, but I think I need it for productive operation to prevent wordpress attacks.

I use Apache on CL7 on fast metal with SSD raid, running DA 1.6 RC1 - latest beta with CB 2359


Details:
Code: 
top - 09:39:04 up 96 days, 18:11,  2 users,  load average: 27.11, 25.17, 24.21
Tasks: 391 total,   8 running, 383 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.3 us, 98.5 sy,  0.2 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 65775276 total,  1452232 free,  1472936 used, 62850108 buff/cache
KiB Swap:  8388604 total,  4185852 free,  4202752 used. 59621940 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
5757 apache    20   0 5840996  80896   3640 S 700.0  0.1   1236:10 /usr/sbin/httpd -DFOREGROUND
24850 apache    20   0 7478944   1140   1140 S 621.9  0.0   6241:31 /usr/sbin/httpd -DFOREGROUND
6145 apache    20   0 5906536  86712   3492 S 615.6  0.1   1603:05 /usr/sbin/httpd -DFOREGROUND
  148 root      20   0       0      0      0 R  90.6  0.0   5780:06 [kswapd0]
  149 root      20   0       0      0      0 R  81.2  0.0   5781:37 [kswapd1]
6757 root      20   0   58704   2352   1484 R  28.1  0.0   0:00.22 /usr/bin/top -c -b -n 1
16582 root      30  10  356712  46256   2432 R   6.2  0.1   9:26.42 /opt/alt/python27/bin/python2.7 /usr/share/lve-stats/lvestats-server.py start --pidfile /var/run/lvestats.pid
4314 root      20   0       0      0      0 D   3.1  0.0   0:00.83 [kworker/3:2]
16559 root      30  10  486548   5312    964 S   3.1  0.0   5:11.28 /opt/alt/python27/bin/python2.7 /usr/share/lve-stats/lvestats-server.py start --pidfile /var/run/lvestats.pid
    1 root      20   0  191392   2600   1320 S   0.0  0.0  42:10.73 /usr/lib/systemd/systemd --system --deserialize 17
    2 root      20   0       0      0      0 S   0.0  0.0   0:13.49 [kthreadd]
    3 root      20   0       0      0      0 S   0.0  0.0   2:49.63 [ksoftirqd/0]
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 [kworker/0:0H]
    8 root      rt   0       0      0      0 S   0.0  0.0   5:32.65 [migration/0]
    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 [rcu_bh]
   10 root      20   0       0      0      0 R   0.0  0.0 176:24.15 [rcu_sched]
   11 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 [lru-add-drain]
   12 root      rt   0       0      0      0 S   0.0  0.0   0:41.27 [watchdog/0]
   13 root      rt   0       0      0      0 S   0.0  0.0   0:44.58 [watchdog/1]
   14 root      rt   0       0      0      0 S   0.0  0.0   5:22.93 [migration/1]
   15 root      20   0       0      0      0 S   0.0  0.0   0:46.30 [ksoftirqd/1]
   17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 [kworker/1:0H]
   18 root      rt   0       0      0      0 S   0.0  0.0   0:35.14 [watchdog/2]
 
Last edited:
HOLY!

Is that 257TB (as in terabytes) for the apache-ip.dir file?

Is that 515PB (as in petabytes... or 1000 terabytes) for the apache-ip.pag?

If so... that's definitely your problem. I'm still in awe that they are that large.
 
I don't think so, it must be a mistake, the system only has a few TBs of space. I wonder how these values come about, I've never seen anything like this. Something is very wrong here, but I have no idea what.

Modsec killed my brand new samsung enterprise SSDs in raid within 3 weeks (TBW).

This is my only DA server at the moment, I'm still testing for the mass migration of CP 2 DA.

I would be interested to know if this problem also occurs with others, I can't imagine that I am the only one with it (the ruleset and apache config has never been changed). I have to stop this behaviour on the productive systems or it will be expensive (SSDs).
 
modsec-sdbm-util could be used for regular cleanup, but I’d suggest switching to OWASP ruleset, if Comodo is causing problems.
 
What does just:

ls -al /tmp/apache-ip.pag /tmp/apache-ip.dir

show?
 
this:
Code: 
[root@da-dev1 ~]# ls -al /tmp/apache-ip.pag /tmp/apache-ip.dir
-rw-r----- 1 apache apache    282261082030080 Jan  7 13:10 /tmp/apache-ip.dir
-rw-r----- 1 apache apache 578721382714580992 Jan  7 15:05 /tmp/apache-ip.pag

we had OWASP on our CP servers, many problems and false positives, the comodo rulset works great on dozens of CP servers
 
How large are the disk?

What does

df -h

show?

I mean ... 282261082030080 bytes translates to about 282 terabytes

578721382714580992 bytes translates to about 578 petabytes.

I really doubt you have this much disk space. I'd almost say you've got some hard drive issues going on. It's not reporting disk usage correctly. And if it's not reporting disk usage correctly...
 
The raid has almost 2 tb, that's why I already said at post #3 that it must be some kind of bug.

I have rebooted the server a few times, put new firmware on the controller and also UEFI and everything else is up to date including SSD firmware.

We use exactly the same server model with the same config with CP, we have dozens of them, no other server has the same problem, all with cloudlinux 7 (same kernel as the da-dev machine).

As soon as I activate the modsec module again and restart apache it pulls up all cores and the file "grows" further.

My goal is to install the perfect DA server, then we clone it and migrate the customers. I could make a clone but I am sure it will behave the same way.

I don't understand what causes this "pseudo growth" of the files and why the apache processes eats the cpus.

The funny thing is that I have these crazy sizes only in the listening itself, otherwise not:

Code: 
[root@da-dev1 ~]# ls -laht /tmp/
total 5.5M
drwxrwxrwt. 35 root    root    4.0K Jan  8 22:05 .
dr-xr-xr-x. 17 root    root     251 Jan  8 21:17 ..
drwx------   3 root    root      17 Jan  8 09:35 systemd-private-f64f49c8a5ed4e1ba0a564a428229d7e-exim.service-bK8jvm
-rw-r-----   1 apache  apache  515P Jan  7 15:05 apache-ip.pag
-rw-r-----   1 apache  apache  257T Jan  7 13:10 apache-ip.dir
drwx------   3 root    root      17 Jan  7 10:15 systemd-private-f64f49c8a5ed4e1ba0a564a428229d7e-named.service-HgfGiC
drwx------   2 root    root      29 Jan  7 10:14 pymp-DV3LSs
drwx------   3 root    root      17 Jan  7 10:14 systemd-private-f64f49c8a5ed4e1ba0a564a428229d7e-freshclam.service-SOSjjq
drwx------   3 root    root      17 Jan  7 10:14 systemd-private-f64f49c8a5ed4e1ba0a564a428229d7e-mysqld.service-0zCQsn
drwx------   3 root    root      17 Jan  7 10:14 systemd-private-f64f49c8a5ed4e1ba0a564a428229d7e-chronyd.service-G01rGr
-rw-r--r--   1 root    root      17 Jan  7 09:40 cwaf_cookies.tmp
-rw-r-----   1 apache  apache  8.0K Dec 30 21:08 apache-default_SESSION.pag
drwx------   2 root    root      29 Dec 30 14:07 pymp-E15xKq
drwx------   2 root    root      29 Dec 12 00:12 pymp-wWL5B1
drwx------   2 root    root      29 Dec 12 00:10 pymp-Ik3h0M
drwx------   2 root    root      29 Dec 12 00:09 pymp-TKtkMI
drwx------   2 root    root      29 Nov 15 11:08 pymp-tyxEg5
drwx------   2 root    root      29 Nov 15 11:06 pymp-w8Zs91
drwx------   2 root    root      29 Nov 15 10:59 pymp-yz1pZB
-rw-------   1 root    root    272K Nov 15 10:54 yum_save_tx.2019-11-15.10-54.Mvt5eW.yumtx
-rw-r--r--   1 root    root    6.8K Oct 31 11:51 cwaf_install.log.29475
.....


[root@da-dev1 ~]# du -d0 -h /tmp/
5.8M    /tmp/
 
We had the same problem on our cpanel servers during the summer. cpanel opened a case for it, CPANEL-27451.

We switched to the owasp rules. But unfortunately there are too many false positives for us. We have really tried to mitigate the rules, but since our customers started to lose trust of us, we switched back to comodo. cpanel had closed CPANEL-27451 (https://docs.cpanel.net/changelogs/84-change-log/). But suddenly our servers started to be unresponsive about every 4th day and reboot was the only solution. This was not caused by high load, what caused it we dont know, we basically gave up on modsec on cpanel (and since then, no server issues).

On our directadmin servers we run the comodo rules with no issues, however these servers are not as loaded as our cpanel servers.

For us, we dont trust neither owasp nor comodo rules for our business. We have not yet tried malware.expert rules, but I am not sure modsec with these comprehensive complex rules is the right way to go. We use modsec now, but only with a few custom rules we understand ourselves to 100%.

But, N.B., regarding owasp, what I understand there is no development on modsec 2.9.x rules, only on 3.x rules. On both our cpanel and da servers we have modsec2. Maybe owasp on modsec3 works better.
 
Thanks for your answer. I'm just trying to find someone who can solve my problem for me for money. We are running on some cpanel servers malware.expert rules, runs great, but I don't like that you can't see the rules. We still need Comodo because it stops thousands of requests, with fine tuning we have almost zero false positives. In DA we lack the modsec log viewer from cpanel with the search functions.

malware.expert rules are too expensive with many servers if there are only 2-3 customers on them
 
And if I dont recall wrong, this is how I installed the parser

Code: 
git clone https://github.com/molu8bits/modsecurity-parser.git
python3 -m pip install -U matplotlib
python3 -m pip install -U pandas
python3 -m pip install -U openpyxl
python3 -m pip install -U Pillow

And this is a cron file we run every hour.

Bash: 
#!/bin/sh
#
# Skapa analysfiler, förslagsvis varje timme.
#
# Rensa gamla analysfiler
#
#####################################################################################
origlogf=/var/log/httpd/modsec_audit.log
logf=/var/log/httpd/modsec_audit.log2
datadir=/var/log/httpd/modsec_output
cronlogf=./cron.log

excelfile=modsec_audit_`date +%Y-%m-%d`.xlsx

cd /usr/local/mparser || exit

echo ============================================================== >> $cronlogf
date >> $cronlogf

test -f $origlogf || exit
test -f modsecurity-parser.py || exit

(
rm -f $excelfile

grep -v "Failed to lock global mutex" $origlogf > $logf

python3 ./modsecurity-parser.py -f $logf -x $excelfile

# Save modsec log file for other analyze tools
cp -af $logf $datadir/

# clean
find $datadir -type f -mtime +1 |
grep -v xlsx |
while read file
do
    rm -f $file
done
) >> $cronlogf 2>&1

# Truncate cron logfile
tail -1000 $cronlogf > $$
cat $$ > $cronlogf
rm -f $$
 
Thanks, I'll try this out on my DA dev server today.

I'll open a feature request for a ModeSec logviewer, I'll send u the link as soon as I created it, would be nice if u can also vote for it
 
To anybody else on this thread...

Is nobody else concerned that the apache-ip.pag and apache-ip.dir files are being reported as woefully large?
 
I know it's an old topic, but I haven't been able to find a solution for 1 month.

I'm having problems with Comodo waf rules and modsec. Ram gets full and Apache crashes. Everything is fine when Apache is restarted, but when it crashes while sleeping, the situation is bad :)

root@:~# ls -alh /tmp/apache-ip.pag /tmp/apache-ip.dir
-rw-r----- 1 apache apache 12M Nov 11 09:45 /tmp/apache-ip.dir
-rw-r----- 1 apache apache 618G Nov 11 16:16 /tmp/apache-ip.pag
Click to expand...
How can I safely delete apache-ip.dir and apache-ip.pag files?
Does rm -rf /tmp/apache-ip.pag cause a problem and apache crashes?

Also currently SecRuleEngine is Off on the Directadmin > Modsecurity page, and Off on the Directadmin >> Comodo WAF 2.24.5 >> Security Engine page. But modsecurity is still running and I see new records on the audit.log page. (I also see CWAF plugin version 2.24.5 Connection error:

How can I stop Modsecurity? (I don't want to remove it completely yet)

Thank you for your help.
 
I don't often look at modsecurity or cwaf but worth to notice I have the same connection error on my server:


Current rules version1.233 (Connection error: )
CWAF plugin version2.24.5 (Connection error: )
 
Hi! So few years later this exact problem still persists on my server. Has anyone found any solution or at least direction how to tackle it?

To reiterate the issue at hand: server with 0 load, with mod_security enabled (comodo / owasp), after one lousy request, cpu usage goes to 100% until httpd is restarted.
 
7669h said:
Hi! So few years later this exact problem still persists on my server. Has anyone found any solution or at least direction how to tackle it?

To reiterate the issue at hand: server with 0 load, with mod_security enabled (comodo / owasp), after one lousy request, cpu usage goes to 100% until httpd is restarted.
Click to expand...
Make a backup before applying what is written here.

I changed the names of apache-ip.dir and apache-ip.pag in the /tmp directory. I restarted httpd.

Apache no longer crashes. And apache-ip.dir and apache-ip.pag were not automatically recreated. Comodo ruleset is active. Modsec is working.

Weird solution but it works for me :)
 
You must log in or register to reply here.
Back
Top