Infrastructure/ansible
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

wiki: 404 a list of bots

Browse source 
Bots are hitting the wiki pretty hard and we don't particularly
care about indexing it anymore, as most real docs should have moved
to docs.fedoraproject.org. Also, many of these bots ignore robots.txt
or do other things we don't want.

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This commit is contained in:
Kevin Fenzi 2025-03-12 10:07:07 -07:00 • committed by kevin
parent e987aa7267
commit af2cadbda9
1 changed files with 5 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

5
roles/httpd/reverseproxy/templates/reversepassproxy.fp-wiki.conf
View file

@ -8,6 +8,11 @@ RewriteCond %{HTTPS} off
RewriteCond %{QUERY_STRING} action= [NC]
RewriteRule .* https://%{HTTP_HOST}%{REQUEST_URI} [R=301,NE,L]
# Spiders-gone-wild
# These spiders may not follow robots.txt and will
# hit admin sections which consume large amounts of CPU
RewriteCond %{HTTP_USER_AGENT} ^.*(Bytespider|ClaudeBot|Amazonbot|YandexBot|ChatGLM-Spider|GPTBot|Barkrowler|YisouSpider|MJ12bot|SemrushBot).*$ [NC]
RewriteRule .* - [F,L]
# /wiki must come before /w due to prefix matching.
ProxyPass {{wikipath}} {{proxyurl}}{{wikipath}}