infrastructure/ansible
1
0
Fork 0
Code Pull requests 7 Activity Actions

Add more spiders which do not seem to honour robots.txt

Browse source 
I went through the last couple of logs afer the first round of 'turn
off the spiders' went out. I looked at the areas which the /robots.txt
disregard and then looked for the bots which ignored it and still
looked up stuff in 'accounts'. This may cut down CPU spikes as these
are looking at dynamic data which can 'blow' things up.

It might be good to add similar tooling to pagure and src since they
seem to be hit a lot in the logs also.

Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>
This commit is contained in:
Stephen Smoogen 2024-07-08 10:12:02 -04:00 • committed by zlopez
parent 377e83fdd1
commit 7e426dbf37
2 changed files with 7 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

3
roles/httpd/website/templates/robots/lists.fedoraproject.org-robots.txt
View file

@ -9,3 +9,6 @@ Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Barkrowler
Disallow: /

6
roles/mailman3/templates/mailmanweb.conf.j2
View file

@ -33,9 +33,11 @@ ProxyPassReverse / http://127.0.0.1:8000/
# Redirecting to hyperkitty if nothing is specified
RewriteEngine on
RewriteRule ^/$ /archives [R,L]
# Spiders-gone-wild
# These spiders do not follow robots.txt
RewriteCond %{HTTP_USER_AGENT} ^.*(Bytespider|ClaudeBot).*$ [NC]
# These spiders may not follow robots.txt and will
# hit admin sections which consume large amounts of CPU
RewriteCond %{HTTP_USER_AGENT} ^.*(Bytespider|ClaudeBot|Amazonbot|YandexBot|claudebot|ChatGLM-Spider|GPTBot|Barkrowler|YisouSpider|MJ12bot).*$ [NC]
RewriteRule .* - [F,L]
# Old static archives