Problem with Custom Httpd.conf

[caneroner]

Hello,

i want to disable some crawler access to some of websites on my server so i tried to write a custom httpd.conf rule below;

SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
SetEnvIfNoCase User-Agent "^Baiduspider" bad_bot
SetEnvIfNoCase User-Agent "^Sogou" bad_bot

Order allow,deny
Allow from all

Deny from env=bad_bots

and after saving it to custom httpd.conf apache gives this error:

Syntax error on line 22 of /usr/local/directadmin/data/users/xxxxxx/httpd.conf:
order not allowed here

line 22 is "Order allow,deny" so, there are some tokens to use in this field but i'm not familiar with this custom httpd.conf settings, any help will be appreciated. thank you.

 

[Arieh]

It seems to me that this should be placed inside a .htaccess file in the public_html of the domain you want it, or am I missing something.

 

[caneroner]

yes it's .htaccss format i guess, so if it's doable from custom httpd.conf i want to know how to do that. Going every site with ftp and modify their .htaccess takes too long...

 

[caneroner]

i think i did it, below code works perfect:

SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot
SetEnvIfNoCase User-Agent .*exabot.* bad_bot
SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot
SetEnvIfNoCase User-Agent .*gigabot.* bad_bot
SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
SetEnvIfNoCase User-Agent "^Baiduspider" bad_bot
SetEnvIfNoCase User-Agent "^Sogou" bad_bot
<Directory /home/user/domains/domain.com/public_html>
Order allow,deny
Allow from all
Deny from env=bad_bot
</Directory>

 

[scsi]

Pretty pointless considering user-agent can be spoofed so easy.

 

[IT_Architect]

Going every site with ftp and modify their .htaccess takes too long...

i think i did it, below code works perfect:

You still need to do it for every domain on your server.

The larger question is how well did it work for you?