Infrastructure/ansible
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

Add more spiders which do not seem to honour robots.txt #2135

Merged
ryanlerch merged 1 commit from spiders-gone-wild-20240708 into main 2024-07-09 09:20:31 +00:00
Conversation 7 Commits 1 Files changed 1 +5
1 changed files with 5 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

5
roles/nagios_server/templates/httpd/nagios.conf.j2
View file

@ -51,6 +51,11 @@ Alias /robots.txt /var/www/robots.txt
RewriteEngine on
RewriteRule ^/$ /nagios/ [R]
# Spiders-gone-wild
# These spiders may not follow robots.txt and will
# hit admin sections which consume large amounts of CPU
RewriteCond %{HTTP_USER_AGENT} ^.*(Bytespider|ClaudeBot|Amazonbot|YandexBot|ChatGLM-Spider|GPTBot|Barkrowler|YisouSpider|MJ12bot).*$ [NC]
RewriteRule .* - [F,L]
Alias /nagios /usr/share/nagios/html/
<Directory "/usr/share/nagios/html">