Infrastructure/ansible
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

Attempt to stop various spiders which do not follow rules.

Browse source 
The ByteSpider and ClaudeBot do not follow robots.txt so need to be
dealt with in other ways.

Signed-off-by: Stephen Smoogen <ssmoogen@redhat.com>
This commit is contained in:
Stephen Smoogen 2024-07-03 17:18:06 +00:00
parent 643e5ae324
commit 4937c5be2a
1 changed files with 4 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

4
roles/mailman3/templates/mailmanweb.conf.j2
View file

@ -33,6 +33,10 @@ ProxyPassReverse / http://127.0.0.1:8000/
# Redirecting to hyperkitty if nothing is specified
RewriteEngine on
RewriteRule ^/$ /archives [R,L]
# Spiders-gone-wild
# These spiders do not follow robots.txt
RewriteCond %{HTTP_USER_AGENT ^.*(Bytespider|ClaudeBot).*$ [NC]
RewriteRule .* - [F,L]
# Old static archives