Block IP addresses which overload server

zEitEr said:
The simplest solution would probably be to deny access to those URLs server-wide, using Location or LocationMatch in Apache/Nginx, or ModRewrite in Apache/LiteSpeed/OpenLiteSpeed.
Click to expand...
The problem is that this list isn't exhaustive.. there are constantly new tries. The only option is scanning logs for 404 / 403 from a single IP and correlate.

This said, I added Firehol Level 1/2/3 to the blocklists and it is awfully quiet out there now.

Code: 
# Firehol Level 1
FIREHOL1|86400|0|https://iplists.firehol.org/files/firehol_level1.netset

# Firehol Level 2
FIREHOL2|86400|0|https://iplists.firehol.org/files/firehol_level2.netset

# Firehol Level 3
FIREHOL3|86400|0|https://iplists.firehol.org/files/firehol_level3.netset
 
iworx said:
This said, I added Firehol Level 1/2/3 to the blocklists and it is awfully quiet out there now.
Click to expand...
Good thing. Downside of this is that this blocks complete netblocks, so very big chance also valid ip's are blocked.
I've also seen a complaint from a year ago that Cloudflare ip's are blocked too. Maybe temporarily but well... it's worth a try probably to use one or more of these lists.

So how high do you have set your DENY_IP_LIMIT? Since for example only the level2.netset contains already 27730 unique IPs and list 1 even has 612798976 so over 61 million. Or does csf only counts subnets against the ip limit?
 
iworx said:
The problem is that this list isn't exhaustive.. there are constantly new tries. The only option is scanning logs for 404 / 403 from a single IP and correlate.
Click to expand...

The real problem is that WordPress the most commonly used platform for creating sites is too resource consuming even when responding with HTTP/404 errors. So if you cache HTTP/404 page or create it static, you will see less peaks, I believe.

iworx said:
The only option is scanning logs for 404 / 403 from a single IP and correlate.
Click to expand...

I widely use this method on servers of my customers, and see all the drawbacks of this method. Some bots might request up to 1000 pages within a minute, so by the moment when the script would run, a server load might be already too high. So if you scan logs for such patterns, make sure you use a rate limit in your webserver.
 
Richard G said:
Good thing. Downside of this is that this blocks complete netblocks, so very big chance also valid ip's are blocked.
I've also seen a complaint from a year ago that Cloudflare ip's are blocked too. Maybe temporarily but well... it's worth a try probably to use one or more of these lists.

So how high do you have set your DENY_IP_LIMIT? Since for example only the level2.netset contains already 27730 unique IPs and list 1 even has 612798976 so over 61 million. Or does csf only counts subnets against the ip limit?
Click to expand...
It counts as 1. Must say, it's a lot quieter on the mailserver too. Pretty happy with the result.
 
Richard G said:
Thank you but what is your DENY_IP_LIMIT setting, I'm just curious because I have it at 15k but I think it needs more. I do have 64 GB Ram so that should not be an issue.
Click to expand...
Oh, sorry. I had it set to 1000. After 24 hours only 7 are added to the list.
 
iworx said:
After 24 hours only 7 are added to the list.
Click to expand...
Ah, then I misunderstood the working. I thought such complete blocklist would be used so if a list had 5k ip's then 5k ip's were preventively already blocked. But if I undertand you correctly, it will only block in iptables when it finds such ip somewhere in a logfile?
Then I don't need to worry.

I have 1000 in the temp_limit as this can not be higher, but with ipset installed I use the 15K limit for the deny limit. :)
 
Richard G said:
Ah, then I misunderstood the working. I thought such complete blocklist would be used so if a list had 5k ip's then 5k ip's were preventively already blocked. But if I undertand you correctly, it will only block in iptables when it finds such ip somewhere in a logfile?
Click to expand...
I understand it the other way around. The myriad IP adresses from Firehol are already blocked, as those lists are implemented into CSF, and because of this, he then gets only 7 "own" new IP adresses blocked in his csf-deny.
Richard G said:
I have 1000 in the temp_limit as this can not be higher, but with ipset installed I use the 15K limit for the deny limit. :)
Click to expand...
Years ago, I went to 30k, but the sites were becoming noticable slower, and the bots didnt ran out of new IP adresses. Its a loosing game. I went back to 2k.
 
johannes said:
Years ago, I went to 30k,
Click to expand...
With the temp limit? Or with the deny limit? If I raise the temp limit over 1000 this notice always appears on restart of csf.
*WARNING* DENY_TEMP_IP_LIMIT sanity check. DENY_TEMP_IP_LIMIT = 3000. Recommended range: 10-1000 (Default: 100)
Click to expand...
Which is why I keep that one on 1000 max.

johannes said:
I understand it the other way around. The myriad IP adresses from Firehol are already blocked, as those lists are implemented into CSF,
Click to expand...
That's what I thought too, but now I'm confused. One of those list contain 15381 subnets.
I got a deny_ip_limit of 15000 in csf.conf so if those liste are implemented into CSF, wouldn't this one list already fill my limit completely?

Or does csf only block what is blocked by command and does it do a lookup in the list on connections made, not using the csf.deny?
 
I'm not entirely sure about this, but I suspect the DENY_TEMP_IP_LIMIT is related to rules in the normal iptables chains, while if you have LF_IPSET=1 the number of stored entries can be much higher.
 
Yes but the DENY_IP_LIMIT setting (which can be much higher) is also set in the normal iptables chain. Both also resides in files on the system. The csf.tempban (in another directory) and csf.deny in the csf directory.
Only when using ipset, it's handled better so you can store a lot more but it related to the DENY_IP_LIMIT (so not temp) if I understand correctly.
 
Richard G said:
With the temp limit? Or with the deny limit? If I raise the temp limit over 1000 this notice always appears on restart of csf.

One of those list contain 15381 subnets. I got a deny_ip_limit of 15000 in csf.conf so if those liste are implemented into CSF, wouldn't this one list already fill my limit completely?

Or does csf only block what is blocked by command and does it do a lookup in the list on connections made, not using the csf.deny?
Click to expand...
The real deny limit, not the temp. I cant remember temp limit, think it was either 1000 or 3000.
I dont think that the subnets and IP adresses in the implemented 3.party lists counts again the internal deny limit. I think the csf deny limit is independent from the implemented lists. But am not 100% sure. Maybe somebody other knows here more?
Also I have no idea about the lookups from 3party lists :-/
 
Oke so CSF is only reading the blocklist(s) and then denying, not actually implementing them in the iptables list.
I thought the limit there was the limit which was allowed to be taken in the iptables from CSF. But then I was mistaken about that. That is good news.
 
@Richard G
It's still in iptables/ipset. But ipset is recommended for this, you can read the size by
Code: 
ipset list
IPSET have optimize logics, if only IPTables handle this, even 50k IPs, it's still slow your server.

deny/temp limit not affect to these 3rd blocklist.

For those limit logics, it will rotating by remove the old IP in "csf.deny/csf.tempdeny" and place new IP.
 
Ohm J said:
But ipset is recommended for this,
Click to expand...
I have ipset installed for a couple of years already so that is not my issue.

Ipset has various lists. For ipv4 I got this one at the moment:
Size in memory: 148720
References: 2
Number of entries: 5140
For ipv6 only 23 ip's at the moment.

I also know about the rotating if the csf deny/tempdeny limits are reached.

My only concern was if the 3rd party blocklist entry's would be added c.q. counted to the to the csf.deny limit. But I understand that is not the case, so then I don't need to worry about that.
 
I have to get back at this.
@Hostmavi your example in post #18 does work, but causes a major security risk.

Because when used that way, the "Require all granted" part, causes manually created .htaccess and .htpasswd files not only not to work, but also that they are readable worldwide.

So I will try another method, just wanted to warn others that your example method can cause this security risk.
 
Richard G said:
I have to get back at this.
@Hostmavi your example in post #18 does work, but causes a major security risk.

Because when used that way, the "Require all granted" part, causes manually created .htaccess and .htpasswd files not only not to work, but also that they are readable worldwide.

So I will try another method, just wanted to warn others that your example method can cause this security risk.
Click to expand...

Thanks for this Richard, this solved one issue with the .htpasswd file and protection, but brings back the initial issue with bots.
What is the other method that you are trying? Can we get an hint or an glimpse on that one?
 
stefantriep said:
Can we get an hint or an glimpse on that one?
Click to expand...
Sure.
At this moment I'm using the method which @Ohm J suggested in post #12 in the httpd-incudes.conf file.

But then there are 2 things.
1.) I have a very long line, because I don't know how to split that to get every bot in a single line. So at this moment it looks like this in my case:
Code: 
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} (adscanner|AhrefsBot|aiohttp|Amazonbot|anthropic-ai|Applebot|AspiegelBot|Baiduspider|BLEXBot|BoardReader|Bytespider
|claudebot|Datanyze|DotBot|facebook|facebookexternalhit/1.1|facebookcatalog/1.0|GPTBot|GPTBot/1.2|Kinza|LieBaoFast|MauiBot|Mb2345Browser|meta-external
agent|MicroMessenger|MJ12bot|msnbot-media|msnbot-MM|nbot|Petalbot|spaziodat|SemrushBot|SiteAuditBot|SemrushBot-BA|SemrushBot-SI|SemrushBot-SWA|Semrush
Bot-CT|SemrushBot-BM|SplitSignalBot|SemrushBot-COUB|serpstatbot|Sogou|YandexBot|YandexImages) [NC]
    RewriteRule (.*) - [F,L]
</IfModule>

As you can see they are all on 1 line now and that's giving a bad overview

2.) In your suggestion there were 2 others which I would like to have, and don't know how to implement these in this code:
Code: 
SetEnvIfNoCase  Request_URI ".env" bad_bot
SetEnvIfNoCase  Request_URI "xlmrpc.php" bad_bot

Maybe you or @Ohm J or somebody else has an idea to fix the overview and a way to add the .env and xlmrpc.php too.
 
A little improve for overview from my post #12,
Code: 
<IfModule mod_rewrite.c>
    RewriteEngine On
    
    BrowserMatchNoCase "aiohttp" bad_bot
    BrowserMatchNoCase "zh_CN" bad_bot
    
    SetEnvIfNoCase  Request_URI "\.env" bad_bot
    SetEnvIfNoCase  Request_URI "xlmrpc\.php" bad_bot
    
    RewriteCond %{ENV:BAD_BOT} !^$
    RewriteRule (.*) - [F,L]
</IfModule>


p.s. "BrowserMatchNoCase " and "SetEnvIfNoCase" are "Regex" style, any special characters should start with "\" to ensure it use as normal string.
 
You must log in or register to reply here.
Back
Top