Infrastructure/infra-docs-fpo
1
0
Fork 0
Code Issues 100 Pull requests 6 Projects Releases Packages Wiki Activity Actions

MirrorManager: update the rest of the docs

Browse source 
Signed-off-by: Aurélien Bompard <aurelien@bompard.org>
This commit is contained in:
Aurélien Bompard 2024-08-06 09:07:13 +02:00 • committed by kevin
parent 042c926241
commit e89773e48b
2 changed files with 43 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

8
modules/sysadmin_guide/pages/fedorawebsites.adoc
View file

@ -110,7 +110,7 @@ and checkout to the new branch:
all sidebars, also in labs, spins, and arm.
. Check all Download links and paths in
`$\{PRODUCT}/prerelease/index.html`. You can find all paths in _bapp01_
(`sudo su - mirrormanager first`) or you can look at the downlaod page
or you can look at the downlaod page
http://dl.fedoraproject.org/pub/alt/stage
. Add CHECKSUM files to `static/checksums` and verify that the paths are
correct. The files should be in _sundries01_ and you can query them with:
@ -170,7 +170,7 @@ all sidebars, also in https://spins.fedoraproject.org,
https://labs.fedoraproject.org/ and https://arm.fedoraproject.org.
. Check all Download links and paths in
`$\{PRODUCT}/prerelease/index.html` You can find all paths in _bapp01_
(`sudo su - mirrormanager first`) or you can look at the downlaod page
or you can look at the downlaod page
https://dl.fedoraproject.org/pub/alt/stage
. Add CHECKSUM files to static/checksums and verify that the paths are
correct. The files should be in sundries and you can query them with:
@ -220,8 +220,8 @@ and checkout to the new branch:
`fedora-web git:(master) git checkout -t -b f22 origin/f22`
. Update the global variables Change curr_state for all arches
. Check all Download links and paths in `$\{PRODUCT}/download/index.html`
You can find all paths in _bapp01_ (`sudo su - mirrormanager first`) or you
can look at the downlaod page http://dl.fedoraproject.org/pub/alt/stage
You can find all paths in _bapp01_ or you can look at the downlaod page
http://dl.fedoraproject.org/pub/alt/stage
. Add CHECKSUM files to `static/checksums` and verify that the paths are
correct. The files should be in _sundries01_ and you can query them with:
+

69
modules/sysadmin_guide/pages/mirrormanager.adoc
View file

@ -9,8 +9,7 @@ Owner::
Contact::
#fedora-admin, sysadmin-main, sysadmin-web
Servers::
mm-frontend01, mm-frontend02, mm-frontend-checkin01, mm-backend01,
mm-crawler01, mm-crawler02
Hosted in OpenShift
Mirrorlist Servers::
Docker container on the proxy servers
Purpose::
@ -22,43 +21,42 @@ MirrorManager handles our mirroring system. It keeps track of lists of
valid mirrors and handles handing out metalink URLs to end users to
download packages from.
The backend server (_mm-backend01_) scans the master mirror (NFS mounted
at `/srv`) using the _mm2_update-master-directory-list_ script (_umdl_)
for changes. Changed directories are detected by comparing the ctime to
the value in the database.
Everything runs in OpenShift. There is a cron job to scan the master mirror
(NFS mounted at `/srv`) using the _mm2_update-master-directory-list_ script
(_umdl_) for changes. Changed directories are detected by comparing the ctime
to the value in the database.
The two crawlers (_mm-crawler01_ and _mm-crawler02_) compare the content
on the mirrors with the results from _umdl_ using RSYNC, HTTP, HTTPS.
The crawler process on _mm-crawler01_ starts at 0:00 and 12:00 and at
2:00 and 14:00 on _mm-crawler02_.
There are also jobs to compare the content on the mirrors with the results
from _umdl_ using RSYNC, HTTP, HTTPS.
The crawler schedule can be viewed in the `vars/apps/mirrormanager.yml` file
in Ansible.
If the content on the mirrors is the same as on the master those mirrors
are included in the dynamic metalink/mirrorlist.
Every hour the backend server generates a python pickle which contains
the information about the state of each mirror. This pickle file is used
by the mirrorlist containers on the proxy servers to dynamically
generate the metalink/mirrorlist for each client individually.
A hourly job generates a binary file which contains the information about the
state of each mirror. This file is used by the mirrorlist containers on the
proxy servers to dynamically generate the metalink/mirrorlist for each client
individually.
The frontend servers (_mm-frontend01_ and _mm-frontend02_) offer an
interface to manipulate the mirrors. Each mirror-admin can only change
the details of the associated mirror. Members of the FAS group
_sysadmin-web_ can seen and change all existing mirrors.
The `frontend` deployment runs the web interface to manipulate the mirrors.
Each mirror-admin can only change the details of the associated mirror.
Members of the FAS group _sysadmin-web_ can seen and change all existing
mirrors.
The mirrorlist provided by the frontend servers has no actively consumed
The mirrorlist provided by the frontend has no actively consumed
content and is therefore heavily cached (12h). It is only used to give
an overview of existing mirrors.
Additionally the frontend servers provide:::
Additionally the frontend provides:::
* an overview of the mirror list usage
https://mirrormanager.fedoraproject.org/statistics
* a propagation overview
https://mirrormanager.fedoraproject.org/propgation
* a mirror map https://mirrormanager.fedoraproject.org/maps
The _mm-frontend-checkin01_ server is only used for _report_mirror_
check-ins. This is used by mirrors to report their status independent of
the crawlers.
The frontend is also used for _report_mirror_ check-ins. This is used by
mirrors to report their status independent of the crawlers.
== Release Preparation
@ -83,7 +81,7 @@ $ rbac-playbook -v /srv/web/infra/ansible/playbooks/manual/mirrormanager/move-to
== mirrorlist containers and mirrorlist servers
Every hour at :55 after the hour, mm-backend01 generates a pkl file with
Every hour at :55 after the hour, a job generates a binary file with
all the current mirrormanager information in it and syncs it to proxies
and mirrorlist-servers. Each proxy accepts requests to
_mirrors.fedoraproject.org_ on apache, then uses haproxy to determine what
@ -106,19 +104,30 @@ the host proxy server.
=== Regenerating the Publiclist
On _mm-backend01_:
On _os-control01_:
....
sudo -u mirrormanager /usr/bin/mm2_update-mirrorlist-server
sudo -u mirrormanager /usr/local/bin/sync_pkl_to_mirrorlists.sh
oc -n mirrormanager create job --from=cj/update-mirrorlist-cache update-mirrorlist-cache-manual
....
Those two commands generates a new mirrorlist pickle and transfers it to
This command generates a new mirrorlist file and transfers it to
the proxies. The mirrorlist containers on the proxies are restarted 15
minutes after each full hour.
The mirrorlist generation can take up to 20 minutes. If a faster
solution is required the mirrorlist pickle from the previous run is
The mirrorlist generation can take up to 20 minutes. The logs can be viewed
with:
....
oc -n mirrormanager logs -f job/update-mirrorlist-cache-manual
....
Once done, the job should be deleted from openshift with:
....
oc -n mirrormanager delete job/update-mirrorlist-cache-manual
....
If a faster solution is required the mirrorlist file from the previous run is
available at:
....