infrastructure/ansible
1
0
Fork 0
Code Pull requests 7 Activity Actions

pagure: drop Bytespider crawler

Browse source 
This crawler is hitting pagure.io really hard.
Sent it a 403 and block in robots.txt

Signed-off-by: Kevin Fenzi <kevin@scrye.com>
This commit is contained in:
Kevin Fenzi 2024-05-01 12:19:43 -07:00
parent 5f2d78a83c
commit 4a3da58a9e
2 changed files with 6 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

4
roles/pagure/templates/0_pagure.conf
View file

@ -138,6 +138,10 @@ MaxConnectionsPerChild 1000
# RewriteEngine On
# RewriteCond %{REQUEST_URI} ^/fedora-web/websites$
# RewriteRule .* - [F]
# Reject Bytespider spider
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} .*Bytespider.*
RewriteRule .* - [F]
<Location /apache-status>
SetHandler server-status

2
roles/pagure/templates/robots.txt.j2
View file

@ -7,4 +7,6 @@ Disallow: /login
Disallow: /*/raw
Disallow: /*/blob
Crawl-Delay: 10
User-agent: Bytespider
Disallow: /
{% endif %}