infrastructure/ansible
1
0
Fork 0
Code Pull requests 7 Activity Actions

lists: block agreessive crawler

Browse source 
This crawler is doing about 2M requests a day to
lists.fedoraproject.org. This is causing db load on db01, causing
services to have issues.

So, block them here for now, but we may want to block them elsewhere
too.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This commit is contained in:
Kevin Fenzi 2024-04-19 11:47:18 -07:00
parent 7e6b1b19f1
commit 6ac1be1f3a
1 changed files with 2 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

2
roles/httpd/website/templates/robots/lists.fedoraproject.org-robots.txt
View file

@ -2,3 +2,5 @@ User-agent: *
Disallow: /accounts/
Disallow: /django-admin/
Disallow: /archives/*/vote$
User-agent: Bytespider
Disallow: /