From aeea2c1d06a68402c6f149a0660083d17fec5184 Mon Sep 17 00:00:00 2001 From: Kevin Fenzi Date: Tue, 10 Oct 2023 17:23:06 -0700 Subject: [PATCH] proxies / koji.fedoraproject.org: disallow robots on koji.fedoraproject.org Turns out back in 2015 (in 623be6ff731) I added a robots.txt on the koji hubs to disallow crawling, but since it's behind the proxies, no one ever saw that robots.txt. Instead they got the default one that just had a 'crawl-delay 1' in it. So, lets tell robots to go away for real now, as they are causing load problems on the koji database and I don't think it's particularly useful for koji to be indexed. It's a lot of dynamic content anyhow. Signed-off-by: Kevin Fenzi --- .../website/templates/robots/koji.fedoraproject.org-robots.txt | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 roles/httpd/website/templates/robots/koji.fedoraproject.org-robots.txt diff --git a/roles/httpd/website/templates/robots/koji.fedoraproject.org-robots.txt b/roles/httpd/website/templates/robots/koji.fedoraproject.org-robots.txt new file mode 100644 index 0000000000..1f53798bb4 --- /dev/null +++ b/roles/httpd/website/templates/robots/koji.fedoraproject.org-robots.txt @@ -0,0 +1,2 @@ +User-agent: * +Disallow: /