ansible/roles/httpd/website/files/robots/robots.txt.admin.fedoraproject.org

User-agent: *
Disallow: /voting
Disallow: /mirrormanager
Disallow: /pkgdb/packages/name
Disallow: /updates

# Temp block http://www.80legs.com/webcrawler.html
User-agent: 008
Disallow: /	

# Block riddler.io
User-agent: Riddler
Disallow: /
Working on the proxies-websites.yml playbook. 2014-12-07 23:39:44 +00:00			`User-agent: *`
			`Disallow: /voting`
			`Disallow: /mirrormanager`
			`Disallow: /pkgdb/packages/name`
Just disallow /updates. Not much point in bodhi updates being in search engines. 2015-04-23 15:01:16 +00:00			`Disallow: /updates`
Working on the proxies-websites.yml playbook. 2014-12-07 23:39:44 +00:00
			`# Temp block http://www.80legs.com/webcrawler.html`
			`User-agent: 008`
			`Disallow: /`
Also tell riddler we don't like them 2015-04-23 14:50:45 +00:00
			`# Block riddler.io`
			`User-agent: Riddler`
			`Disallow: /`