Infrastructure/infra-docs-fpo
1
0
Fork 0
Code Issues 100 Pull requests 6 Projects Releases Packages Wiki Activity Actions

Review mirrormanager SOP

Browse source 
Signed-off-by: Michal Konečný <mkonecny@redhat.com>
This commit is contained in:
Michal Konečný 2021-09-07 13:00:48 +02:00
parent 629824edbc
commit 36a08c9afd
2 changed files with 18 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

2
modules/sysadmin_guide/nav.adoc
View file

@ -71,7 +71,7 @@
** xref:memcached.adoc[Memcached Infrastructure - SOP] ** xref:memcached.adoc[Memcached Infrastructure - SOP]
** xref:message-tagging-service.adoc[Message Tagging Service - SOP] ** xref:message-tagging-service.adoc[Message Tagging Service - SOP]
** xref:mirrorhiding.adoc[Mirror Hiding Infrastructure - SOP] ** xref:mirrorhiding.adoc[Mirror Hiding Infrastructure - SOP]
** xref:mirrormanager.adoc[mirrormanager - SOP in review ] ** xref:mirrormanager.adoc[MirrorManager Infrastructure - SOP]
** xref:mirrormanager-S3-EC2-netblocks.adoc[AWS Mirrors - SOP] ** xref:mirrormanager-S3-EC2-netblocks.adoc[AWS Mirrors - SOP]
** xref:mote.adoc[mote - SOP in review ] ** xref:mote.adoc[mote - SOP in review ]
** xref:nagios.adoc[nagios - SOP in review ] ** xref:nagios.adoc[nagios - SOP in review ]

35
modules/sysadmin_guide/pages/mirrormanager.adoc
View file

@ -8,8 +8,6 @@ Owner::
Fedora Infrastructure Team Fedora Infrastructure Team
Contact:: Contact::
#fedora-admin, sysadmin-main, sysadmin-web #fedora-admin, sysadmin-main, sysadmin-web
Location::
Phoenix
Servers:: Servers::
mm-frontend01, mm-frontend02, mm-frontend-checkin01, mm-backend01, mm-frontend01, mm-frontend02, mm-frontend-checkin01, mm-backend01,
mm-crawler01, mm-crawler02 mm-crawler01, mm-crawler02
@ -25,7 +23,7 @@ valid mirrors and handles handing out metalink URLs to end users to
download packages from. download packages from.
The backend server (_mm-backend01_) scans the master mirror (NFS mounted The backend server (_mm-backend01_) scans the master mirror (NFS mounted
at /srv) using the _mm2_update-master-directory-list_ script (_umdl_) at `/srv`) using the _mm2_update-master-directory-list_ script (_umdl_)
for changes. Changed directories are detected by comparing the ctime to for changes. Changed directories are detected by comparing the ctime to
the value in the database. the value in the database.
@ -65,7 +63,7 @@ the crawlers.
== Release Preparation == Release Preparation
MirrorManager should automatically detect the new release version, and MirrorManager should automatically detect the new release version, and
will create a new Version() object in the database. This is visible on will create a new `Version()` object in the database. This is visible on
the Version page in the web UI, and on the Version page in the web UI, and on
https://admin.fedoraproject.org/mirrormanager/. https://admin.fedoraproject.org/mirrormanager/.
@ -75,8 +73,8 @@ contact the Mirror Wrangler.
== One Week After a Release == One Week After a Release
In the first week after the release MirrorManager still uses the files In the first week after the release MirrorManager still uses the files
at fedora/linux/development/<version> and not at at `fedora/linux/development/<version>` and not at
fedora/linux/releases/<version> `fedora/linux/releases/<version>`
Once enough mirrors have picked up the files in the release directory Once enough mirrors have picked up the files in the release directory
following script (on _mm-backend01_) can be used to change the paths in following script (on _mm-backend01_) can be used to change the paths in
@ -103,20 +101,20 @@ sudo -u mirrormanager mm2_move-to-archive --originalCategory='Fedora EPEL' --dir
Every hour at :55 after the hour, mm-backend01 generates a pkl file with Every hour at :55 after the hour, mm-backend01 generates a pkl file with
all the current mirrormanager information in it and syncs it to proxies all the current mirrormanager information in it and syncs it to proxies
and mirrorlist-servers. Each proxy accepts requests to and mirrorlist-servers. Each proxy accepts requests to
mirrors.fedoraproject.org on apache, then uses haproxy to determine what _mirrors.fedoraproject.org_ on apache, then uses haproxy to determine what
backend will reply. There are 2 containers defined on each proxy: backend will reply. There are 2 containers defined on each proxy:
mirrorlist1 and mirrorlist2. haproxy will look for those first, then mirrorlist1 and mirrorlist2. haproxy will look for those first, then
fall back to any of the mirrorlist servers defined over the vpn. fall back to any of the mirrorlist servers defined over the vpn.
At :15 after the hour, a script runs on all proxies: At :15 after the hour, a script runs on all proxies:
/usr/local/bin/restart-mirrorlist-containers This script starts up `/usr/local/bin/restart-mirrorlist-containers` This script starts up
mirrorlist2 container, makes sure it can process requests and then if mirrorlist2 container, makes sure it can process requests and then if
so, restarts mirrorlist1 container with the new pkl data. If not, so, restarts mirrorlist1 container with the new pkl data. If not,
mirrorlist1 keeps running with the old data. During this process at mirrorlist1 keeps running with the old data. During this process at
least one (with mirrorlists servers as backup) server is processing least one (with mirrorlists servers as backup) server is processing
requests so users see no issues. requests so users see no issues.
mirrorlist-containers log to /var/log/mirrormanager/mirrorlist\{1|2}/ on `mirrorlist-containers` log to `/var/log/mirrormanager/mirrorlist\{1|2}/` on
the host proxy server. the host proxy server.
== Troubleshooting and Resolution == Troubleshooting and Resolution
@ -148,25 +146,26 @@ The container used for mirrorlists is the mirrormanager2-mirrorlist
container in Fedora dist git: container in Fedora dist git:
https://src.fedoraproject.org/cgit/container/mirrormanager2-mirrorlist.git/ https://src.fedoraproject.org/cgit/container/mirrormanager2-mirrorlist.git/
The one being used is defined in a ansible variable in: The one being used is defined in a ansible variable in:
roles/mirrormanager/mirrorlist_proxy/defaults/main.yml and in turn used roles/mirrormanager/mirrorlist_proxy/defaults/main.yml
in systemd unit files for mirrorlist1 and mirrorlist2. To update the (TODO: This file no longer exists, find the new place where this is defined)
and in turn used in systemd unit files for mirrorlist1 and mirrorlist2. To update the
container used, update this variable, run the playbook and then restart container used, update this variable, run the playbook and then restart
the mirrorlist1 and mirrorlist2 containers on each proxy. Note that this the mirrorlist1 and mirrorlist2 containers on each proxy. Note that this
may take a while the first time as the image has to be downloaded from may take a while the first time as the image has to be downloaded from
our registiry. our registry.
=== Debugging problems with mirrorlist container startup === Debugging problems with mirrorlist container startup
Sometimes on boot some hosts won't be properly serving mirrorlists. This Sometimes on boot some hosts won't be properly serving mirrorlists. This
is due to a container startup issue. run: 'docker ps -a' as root to see is due to a container startup issue. run: `docker ps -a` as root to see
the active containers. It will usually say something like 'exited(1)' or the active containers. It will usually say something like 'exited(1)' or
the like. Record the container id and then run: 'docker rm --force the like. Record the container id and then run: `docker rm --force
<containerid>' then run 'docker ps -a' and confirm nothing shows. Then <containerid>` then run `docker ps -a` and confirm nothing shows. Then
run 'systemctl start mirrorlist1' and it should correctly start run `systemctl start mirrorlist1` and it should correctly start
mirrorlist1. mirrorlist1.
=== General debugging for mirrorlist containers === General debugging for mirrorlist containers
docker commands like 'docker ps -a' show a fair bit of information. Docker commands like `docker ps -a` show a fair bit of information.
Also, systemctl status mirrorlist1/2 or the journal should have Also, `systemctl status mirrorlist1/2` or the journal should have
information when a container is failing. information when a container is failing.