Add guide for blocking IPs

This guide describes how to find and block suspicious IPs. Hopefully
this will help us to have unified process when dealing with DDoS or AI scrapers.

modules/howtos/pages/blocking_ip.adoc

@@ -0,0 +1,98 @@
+= How to identify and block suspicious IP
+
+[NOTE]
+====
+You need to have admin access to proxies or the server in question to be able to follow this guide
+====
+
+This guide serves as a guidance what to do if some service is under heavy load because of the
+amount of requests. You can see in link:https://nagios.fedoraproject.org/nagios/[nagios]
+something like `Service not reachable` or `Socket timeout`.
+
+== Blocking IP directly on the service
+
+. Log in to the server in question (for example in case of pagure.io you login
+to pagure02.iad2.fedoraproject.org)
++
+----
+ssh pagure02.iad2.fedoraproject.org
+----
++
+. Switch to root
++
+----
+sudo su
+----
++
+. Run the following on the server
++
+----
+date && cat /var/log/httpd/access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -20
+----
++
+This will print a current time followed by the 20 IPs with most requests sent to the server.
+You will need to run it at least twice to see which number is rising. Those that are rising
+too fast (for example 40 requests per minute) are the ones we want to block.
++
+. Check the IP with `whois` and `nslookup`
++
+----
+whois <ip_address>
+nslookup <ip_address>
+----
++
+If the output of this is not Red Hat, Cloudfront or internal services (most of the services are routed through
+proxies, so they can show up as most active) it's usually OK to block them. Not every server has
+these tool available in this case run it on your machine.
++
+. Block the IP on server
++
+----
+iptables -I INPUT -s <IP> -j REJECT
+----
++
+This will add rules to iptables to REJECT all the requests from the specified IP.
+
+
+== Blocking IPs on proxies
+. Log in to proxy that is being shown up as the most active on the server above or the one that is
+alerting in link:https://nagios.fedoraproject.org/nagios/[nagios]
++
+----
+ssh proxy01.iad2.fedoraproject.org
+----
++
+. Switch to root
++
+----
+sudo su
+----
++
+. Run the following on the server
++
+----
+date && cat /var/log/httpd/fedoraproject.org-access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -20
+----
++
+The proxies have access logs for most of the services in fedora infra. So you need to find the correct
+one. Look for the one that is being reported (for example `src.fedoraproject.org-access.log). Some of them
+like wiki is in `fedoraproject.org-access.log`, so if you don't find the specific one,
+look at this one.
++
+. Check the IP with `whois` and `nslookup`
++
+----
+whois <ip_address>
+nslookup <ip_address>
+----
++
+If the output of this is not Red Hat, Cloudfront or internal services it's usually OK to block them.
+Not every server has these tool available in this case run it on your machine.
++
+. Block the IP on all proxies
++
+----
+ansible proxies -m shell -a "iptables -I INPUT -s <IP> -j REJECT"
+----
++
+This needs to be run on batcave01 and blocks the IP on all machines in `proxies` group in inventory.

modules/sysadmin_guide/pages/index.adoc

@@ -187,6 +187,7 @@ In this section is list of guides for common tasks that are done in Fedora Infra
 * xref:howtos:archive_list.adoc[How to Archive mailing list]
 * xref:howtos:archive-old-fedora.adoc[How to Archive Old Fedora Releases]
 * xref:howtos:add_external_hardware_to_vpn.adoc[Add external servers to vpn]
+* xref:howtos:blocking_ip.adoc[How to identify and block suspicious IP]
 * xref:howtos:check_robosignatory_production_logs.adoc[How to check robosignatory productions logs]
 * xref:howtos:clean_2f_tokens.adoc[How to remove 2 factor authentication tokens in IPA]
 * xref:howtos:clean_monitoring_sidetags.adoc[How to clean up the side-tags created by the monitor-gating project]