How to block robot / spiders with common names

cDGo

Verified User
Joined
Sep 21, 2012
Messages
108
empty user agent string
Unknown robot identified by bot\*
Firefox version 10 and lower - various robots
Windows 99 - ( Rogue Robot )
MSIE 5 - ( Rogue Robot )
MSIE 6 - ( Rogue Robot )
Windows 95 - ( Rogue Robot )

These are some high bandwidth burners.
How do I identify the exact user-agent in order to block them in the htaccess file?
Because they use common names, or none which makes it difficult to block only the bots
 
# Enable Rewrite Engine
RewriteEngine On

# Block empty User-Agent string
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]

# Block bots with "Unknown robot identified by bot*" in User-Agent
RewriteCond %{HTTP_USER_AGENT} ^Unknown\srobot\sidentified\sby\sbot.*$ [NC,OR]

# Block Firefox version 10 and lower
RewriteCond %{HTTP_USER_AGENT} Firefox\/([1-9]|10)(\.|$) [NC,OR]

# Block User-Agents with "Windows 99"
RewriteCond %{HTTP_USER_AGENT} Windows\s?99 [NC,OR]

# Block MSIE 5 and 6
RewriteCond %{HTTP_USER_AGENT} MSIE\s[56]\. [NC,OR]

# Block Windows 95
RewriteCond %{HTTP_USER_AGENT} Windows\s?95 [NC]

# Deny Access
RewriteRule .* - [F,L]
 
Hi,
Thank you, just to be sure:

RewriteCond %{HTTP_USER_AGENT} Firefox\/([1-9]|10)(\.|$) [NC,OR]
Doesn't this block also regular visitors with the firefox browser?
 
RewriteCond %{HTTP_USER_AGENT} Firefox\/([1-9]|10)(\.|$) [NC,OR]
Doesn't this block also regular visitors with the firefox browser?
Yes you are right , was a quick paste because you were also mentioning firefox.
Just remove that line(s) and see if it help
 
Back
Top