From 2140cf2080baa3b2502a9bd878e26109b5e428da Mon Sep 17 00:00:00 2001 From: Kevin Fenzi Date: Wed, 3 Jul 2024 09:58:18 -0700 Subject: [PATCH] mailman: adjust the right robots.txt and fix formatting Signed-off-by: Kevin Fenzi --- .../templates/robots/lists.fedoraproject.org-robots.txt | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/roles/httpd/website/templates/robots/lists.fedoraproject.org-robots.txt b/roles/httpd/website/templates/robots/lists.fedoraproject.org-robots.txt index 8e817aaa29..a51422965a 100644 --- a/roles/httpd/website/templates/robots/lists.fedoraproject.org-robots.txt +++ b/roles/httpd/website/templates/robots/lists.fedoraproject.org-robots.txt @@ -2,5 +2,10 @@ User-agent: * Disallow: /accounts/ Disallow: /django-admin/ Disallow: /archives/*/vote$ +Crawl-Delay: 10 + User-agent: Bytespider Disallow: / + +User-agent: ClaudeBot +Disallow: /