From 36a08c9afdeedbee3b648eb7ef0a6ac8a509a300 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Michal=20Kone=C4=8Dn=C3=BD?= Date: Tue, 7 Sep 2021 13:00:48 +0200 Subject: [PATCH] Review mirrormanager SOP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Michal Konečný --- modules/sysadmin_guide/nav.adoc | 2 +- .../sysadmin_guide/pages/mirrormanager.adoc | 35 +++++++++---------- 2 files changed, 18 insertions(+), 19 deletions(-) diff --git a/modules/sysadmin_guide/nav.adoc b/modules/sysadmin_guide/nav.adoc index addd77e..6e0af19 100644 --- a/modules/sysadmin_guide/nav.adoc +++ b/modules/sysadmin_guide/nav.adoc @@ -71,7 +71,7 @@ ** xref:memcached.adoc[Memcached Infrastructure - SOP] ** xref:message-tagging-service.adoc[Message Tagging Service - SOP] ** xref:mirrorhiding.adoc[Mirror Hiding Infrastructure - SOP] -** xref:mirrormanager.adoc[mirrormanager - SOP in review ] +** xref:mirrormanager.adoc[MirrorManager Infrastructure - SOP] ** xref:mirrormanager-S3-EC2-netblocks.adoc[AWS Mirrors - SOP] ** xref:mote.adoc[mote - SOP in review ] ** xref:nagios.adoc[nagios - SOP in review ] diff --git a/modules/sysadmin_guide/pages/mirrormanager.adoc b/modules/sysadmin_guide/pages/mirrormanager.adoc index 63a8f79..710c472 100644 --- a/modules/sysadmin_guide/pages/mirrormanager.adoc +++ b/modules/sysadmin_guide/pages/mirrormanager.adoc @@ -8,8 +8,6 @@ Owner:: Fedora Infrastructure Team Contact:: #fedora-admin, sysadmin-main, sysadmin-web -Location:: - Phoenix Servers:: mm-frontend01, mm-frontend02, mm-frontend-checkin01, mm-backend01, mm-crawler01, mm-crawler02 @@ -25,7 +23,7 @@ valid mirrors and handles handing out metalink URLs to end users to download packages from. The backend server (_mm-backend01_) scans the master mirror (NFS mounted -at /srv) using the _mm2_update-master-directory-list_ script (_umdl_) +at `/srv`) using the _mm2_update-master-directory-list_ script (_umdl_) for changes. Changed directories are detected by comparing the ctime to the value in the database. @@ -65,7 +63,7 @@ the crawlers. == Release Preparation MirrorManager should automatically detect the new release version, and -will create a new Version() object in the database. This is visible on +will create a new `Version()` object in the database. This is visible on the Version page in the web UI, and on https://admin.fedoraproject.org/mirrormanager/. @@ -75,8 +73,8 @@ contact the Mirror Wrangler. == One Week After a Release In the first week after the release MirrorManager still uses the files -at fedora/linux/development/ and not at -fedora/linux/releases/ +at `fedora/linux/development/` and not at +`fedora/linux/releases/` Once enough mirrors have picked up the files in the release directory following script (on _mm-backend01_) can be used to change the paths in @@ -103,20 +101,20 @@ sudo -u mirrormanager mm2_move-to-archive --originalCategory='Fedora EPEL' --dir Every hour at :55 after the hour, mm-backend01 generates a pkl file with all the current mirrormanager information in it and syncs it to proxies and mirrorlist-servers. Each proxy accepts requests to -mirrors.fedoraproject.org on apache, then uses haproxy to determine what +_mirrors.fedoraproject.org_ on apache, then uses haproxy to determine what backend will reply. There are 2 containers defined on each proxy: mirrorlist1 and mirrorlist2. haproxy will look for those first, then fall back to any of the mirrorlist servers defined over the vpn. At :15 after the hour, a script runs on all proxies: -/usr/local/bin/restart-mirrorlist-containers This script starts up +`/usr/local/bin/restart-mirrorlist-containers` This script starts up mirrorlist2 container, makes sure it can process requests and then if so, restarts mirrorlist1 container with the new pkl data. If not, mirrorlist1 keeps running with the old data. During this process at least one (with mirrorlists servers as backup) server is processing requests so users see no issues. -mirrorlist-containers log to /var/log/mirrormanager/mirrorlist\{1|2}/ on +`mirrorlist-containers` log to `/var/log/mirrormanager/mirrorlist\{1|2}/` on the host proxy server. == Troubleshooting and Resolution @@ -148,25 +146,26 @@ The container used for mirrorlists is the mirrormanager2-mirrorlist container in Fedora dist git: https://src.fedoraproject.org/cgit/container/mirrormanager2-mirrorlist.git/ The one being used is defined in a ansible variable in: -roles/mirrormanager/mirrorlist_proxy/defaults/main.yml and in turn used -in systemd unit files for mirrorlist1 and mirrorlist2. To update the +roles/mirrormanager/mirrorlist_proxy/defaults/main.yml +(TODO: This file no longer exists, find the new place where this is defined) +and in turn used in systemd unit files for mirrorlist1 and mirrorlist2. To update the container used, update this variable, run the playbook and then restart the mirrorlist1 and mirrorlist2 containers on each proxy. Note that this may take a while the first time as the image has to be downloaded from -our registiry. +our registry. === Debugging problems with mirrorlist container startup Sometimes on boot some hosts won't be properly serving mirrorlists. This -is due to a container startup issue. run: 'docker ps -a' as root to see +is due to a container startup issue. run: `docker ps -a` as root to see the active containers. It will usually say something like 'exited(1)' or -the like. Record the container id and then run: 'docker rm --force -' then run 'docker ps -a' and confirm nothing shows. Then -run 'systemctl start mirrorlist1' and it should correctly start +the like. Record the container id and then run: `docker rm --force +` then run `docker ps -a` and confirm nothing shows. Then +run `systemctl start mirrorlist1` and it should correctly start mirrorlist1. === General debugging for mirrorlist containers -docker commands like 'docker ps -a' show a fair bit of information. -Also, systemctl status mirrorlist1/2 or the journal should have +Docker commands like `docker ps -a` show a fair bit of information. +Also, `systemctl status mirrorlist1/2` or the journal should have information when a container is failing.