Infrastructure/ansible
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

mailman: adjust the right robots.txt and fix formatting

Browse source 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This commit is contained in:
Kevin Fenzi 2024-07-03 09:58:18 -07:00
parent 536e0a71b0
commit 2140cf2080
1 changed files with 5 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

5
roles/httpd/website/templates/robots/lists.fedoraproject.org-robots.txt
View file

@ -2,5 +2,10 @@ User-agent: *
Disallow: /accounts/
Disallow: /django-admin/
Disallow: /archives/*/vote$
Crawl-Delay: 10
User-agent: Bytespider
Disallow: /
User-agent: ClaudeBot
Disallow: /