diff --git a/modules/howtos/pages/blocking_ip.adoc b/modules/howtos/pages/blocking_ip.adoc new file mode 100644 index 0000000..03bada9 --- /dev/null +++ b/modules/howtos/pages/blocking_ip.adoc @@ -0,0 +1,98 @@ += How to identify and block suspicious IP + +[NOTE] +==== +You need to have admin access to proxies or the server in question to be able to follow this guide +==== + +This guide serves as a guidance what to do if some service is under heavy load because of the +amount of requests. You can see in link:https://nagios.fedoraproject.org/nagios/[nagios] +something like `Service not reachable` or `Socket timeout`. + +== Blocking IP directly on the service + +. Log in to the server in question (for example in case of pagure.io you login +to pagure02.iad2.fedoraproject.org) ++ +---- +ssh pagure02.iad2.fedoraproject.org +---- ++ +. Switch to root ++ +---- +sudo su +---- ++ +. Run the following on the server ++ +---- +date && cat /var/log/httpd/access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -20 +---- ++ +This will print a current time followed by the 20 IPs with most requests sent to the server. +You will need to run it at least twice to see which number is rising. Those that are rising +too fast (for example 40 requests per minute) are the ones we want to block. ++ +. Check the IP with `whois` and `nslookup` ++ +---- +whois +nslookup +---- ++ +If the output of this is not Red Hat, Cloudfront or internal services (most of the services are routed through +proxies, so they can show up as most active) it's usually OK to block them. Not every server has +these tool available in this case run it on your machine. ++ +. Block the IP on server ++ +---- +iptables -I INPUT -s -j REJECT +---- ++ +This will add rules to iptables to REJECT all the requests from the specified IP. + + +== Blocking IPs on proxies +. Log in to proxy that is being shown up as the most active on the server above or the one that is +alerting in link:https://nagios.fedoraproject.org/nagios/[nagios] ++ +---- +ssh proxy01.iad2.fedoraproject.org +---- ++ +. Switch to root ++ +---- +sudo su +---- ++ +. Run the following on the server ++ +---- +date && cat /var/log/httpd/fedoraproject.org-access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -20 +---- ++ +The proxies have access logs for most of the services in fedora infra. So you need to find the correct +one. Look for the one that is being reported (for example `src.fedoraproject.org-access.log). Some of them +like wiki is in `fedoraproject.org-access.log`, so if you don't find the specific one, +look at this one. ++ +. Check the IP with `whois` and `nslookup` ++ +---- +whois +nslookup +---- ++ +If the output of this is not Red Hat, Cloudfront or internal services it's usually OK to block them. +Not every server has these tool available in this case run it on your machine. ++ +. Block the IP on all proxies ++ +---- +ansible proxies -m shell -a "iptables -I INPUT -s -j REJECT" +---- ++ +This needs to be run on batcave01 and blocks the IP on all machines in `proxies` group in inventory. diff --git a/modules/sysadmin_guide/pages/index.adoc b/modules/sysadmin_guide/pages/index.adoc index f1f358e..d5e6d59 100644 --- a/modules/sysadmin_guide/pages/index.adoc +++ b/modules/sysadmin_guide/pages/index.adoc @@ -187,6 +187,7 @@ In this section is list of guides for common tasks that are done in Fedora Infra * xref:howtos:archive_list.adoc[How to Archive mailing list] * xref:howtos:archive-old-fedora.adoc[How to Archive Old Fedora Releases] * xref:howtos:add_external_hardware_to_vpn.adoc[Add external servers to vpn] +* xref:howtos:blocking_ip.adoc[How to identify and block suspicious IP] * xref:howtos:check_robosignatory_production_logs.adoc[How to check robosignatory productions logs] * xref:howtos:clean_2f_tokens.adoc[How to remove 2 factor authentication tokens in IPA] * xref:howtos:clean_monitoring_sidetags.adoc[How to clean up the side-tags created by the monitor-gating project]