My server was sending out high usage alerts. Upon investigation I could see that an SEO company is crawling through blog posts at a high rate.
The blog post URLs have a structure like: /tag/keyword/
The requests were all coming from 99.888.777.X (I've replaced the real IP address)
So here's what I did:
Now, I understand that requests from these IP addresses can still show up in my httpd logs.
The request is still made, and the traffic is allegedly being blocked thereafter.
But what I'm having trouble understanding is why can I still see MySQL queries that are undoubtedly being triggered by requests from 99.888.777.X ?
Should that be possible?
I know the MySQL queries are coming from this IP address because...
a) I watch for all of the live MySQL queries being made, and when I see a query being generated for a very obscure keyword...
b) I then immediately launch the following command:
tail -n 1000 /var/log/httpd/domains/domain.com.log | awk '{print $4, $1, $7}' | sort -r | head -n 50
c) And I can see the request for the URL with the same obscure keyword being made by 99.888.777.X within the last 10 seconds
[08/Jan/2025:13:28:24 99.888.777.19 /tag/kurumon-ichiba-market
I do use Cloudflare.
Thanks for any insight anyone can provide.
The blog post URLs have a structure like: /tag/keyword/
The requests were all coming from 99.888.777.X (I've replaced the real IP address)
So here's what I did:
# csf -d 99.888.777.0/24
# nano /etc/csf/csf.deny
Added - Do not delete -
to the comments beside the IP address
# csf -r
# lfd -r
# csf -g 99.888.777.0/24
^^ shows:
Table Chain num pkts bytes target prot opt in out source destination
filter DENYIN 989 0 0 DROP all -- !lo * 99.888.777.0/24 0.0.0.0/0
filter DENYOUT 989 0 0 LOGDROPOUT all -- * !lo 0.0.0.0/0 99.888.777.0/24
# nano /etc/csf/csf.deny
^^ shows:
99.888.777.0/24 # - Do not delete - Manually denied: 99.888.777.0/24 (SG/Singapore/-) - Mon Jan 6 15:40:44 2025
Now, I understand that requests from these IP addresses can still show up in my httpd logs.
The request is still made, and the traffic is allegedly being blocked thereafter.
But what I'm having trouble understanding is why can I still see MySQL queries that are undoubtedly being triggered by requests from 99.888.777.X ?
Should that be possible?
I know the MySQL queries are coming from this IP address because...
a) I watch for all of the live MySQL queries being made, and when I see a query being generated for a very obscure keyword...
b) I then immediately launch the following command:
tail -n 1000 /var/log/httpd/domains/domain.com.log | awk '{print $4, $1, $7}' | sort -r | head -n 50
c) And I can see the request for the URL with the same obscure keyword being made by 99.888.777.X within the last 10 seconds
[08/Jan/2025:13:28:24 99.888.777.19 /tag/kurumon-ichiba-market
I do use Cloudflare.
Thanks for any insight anyone can provide.