hi to all,
few days ago I have alot of traffic from this badbot, and I find several way to block, with .htaccess, and robot.txt, but I always preferd iptables, so I install fail2ban and here is the conf to block this bot and few another
test in Centos 6.7, 64b:
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
yum install fail2ban
chkconfig fail2ban --level=235 on
cd /etc/fail2ban/filter.d
vi apache-badbots.conf
add this before "sogou music spider"
|tools\.ua\.random|Mozilla/5\.0 \(compatible; MJ12bot/v1\.4\.5; http\://www\.majestic12\.co\.uk/bot\.php\?\+\)
ex:
badbotscustom = EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|tools\.ua\.random|Mozilla/5\.0 \(compatible; MJ12bot/v1\.4\.5; http\://www\.majestic12\.co\.uk/bot\.php\?\+\)
vi /etc/fail2ban/jail.conf
modified all [apache-badbots] with this:
[apache-badbots]
enabled = true
filter = apache-badbots
# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
port = http,https
logpath = /var/log/httpd/domains/*.log
bantime = 172800
maxretry = 3
save the file, and:
service fail2ban start
you can see when ip of this bot is blocked here:
cat /var/log/messages | grep fail2ban|grep Found
bye
few days ago I have alot of traffic from this badbot, and I find several way to block, with .htaccess, and robot.txt, but I always preferd iptables, so I install fail2ban and here is the conf to block this bot and few another
test in Centos 6.7, 64b:
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
yum install fail2ban
chkconfig fail2ban --level=235 on
cd /etc/fail2ban/filter.d
vi apache-badbots.conf
add this before "sogou music spider"
|tools\.ua\.random|Mozilla/5\.0 \(compatible; MJ12bot/v1\.4\.5; http\://www\.majestic12\.co\.uk/bot\.php\?\+\)
ex:
badbotscustom = EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|tools\.ua\.random|Mozilla/5\.0 \(compatible; MJ12bot/v1\.4\.5; http\://www\.majestic12\.co\.uk/bot\.php\?\+\)
vi /etc/fail2ban/jail.conf
modified all [apache-badbots] with this:
[apache-badbots]
enabled = true
filter = apache-badbots
# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
port = http,https
logpath = /var/log/httpd/domains/*.log
bantime = 172800
maxretry = 3
save the file, and:
service fail2ban start
you can see when ip of this bot is blocked here:
cat /var/log/messages | grep fail2ban|grep Found
bye