Infrastructure/ansible
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

proxies / koji.fedoraproject.org: disallow robots on koji.fedoraproject.org

Browse source 
Turns out back in 2015 (in 623be6ff73) I added a robots.txt on the koji
hubs to disallow crawling, but since it's behind the proxies, no one
ever saw that robots.txt. Instead they got the default one that just had
a 'crawl-delay 1' in it.

So, lets tell robots to go away for real now, as they are causing load
problems on the koji database and I don't think it's particularly useful
for koji to be indexed. It's a lot of dynamic content anyhow.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This commit is contained in:
Kevin Fenzi 2023-10-10 17:23:06 -07:00 • committed by kevin
parent b0ddd09187
commit aeea2c1d06
1 changed files with 2 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

2
roles/httpd/website/templates/robots/koji.fedoraproject.org-robots.txt Normal file
View file

@ -0,0 +1,2 @@
User-agent: *
Disallow: /