Infrastructure/ansible
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

src: adjust robots.txt and fix formatting

Browse source 
Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This commit is contained in:
Kevin Fenzi 2024-07-03 10:00:01 -07:00
parent 2140cf2080
commit 6c2f16eb9a
1 changed files with 4 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

4
roles/httpd/website/templates/robots/src.fedoraproject.org-robots.txt
View file

@ -1,4 +1,8 @@
User-agent: *
Crawl-delay: 5
User-agent: Bytespider
Disallow: /
User-agent: ClaudeBot
Disallow: /