From e5334de24045ab5218f33f1ddaf9213911c8046f Mon Sep 17 00:00:00 2001 From: Nils Philippsen Date: Wed, 12 Jul 2023 15:02:43 +0200 Subject: [PATCH] Update DNF Counting SOP for new mirrors-countme Fixes: https://github.com/fedora-infra/mirrors-countme/issues/84 Signed-off-by: Nils Philippsen --- .../sysadmin_guide/pages/dnf-counting.adoc | 70 +++++++++++++------ 1 file changed, 48 insertions(+), 22 deletions(-) diff --git a/modules/sysadmin_guide/pages/dnf-counting.adoc b/modules/sysadmin_guide/pages/dnf-counting.adoc index b5c48b4..4c0563f 100644 --- a/modules/sysadmin_guide/pages/dnf-counting.adoc +++ b/modules/sysadmin_guide/pages/dnf-counting.adoc @@ -8,17 +8,15 @@ installations. Owner:: Fedora Infrastructure Team Contact:: - #fedora-admin, #fedora-noc -Initiative Representatives:: - * Nils Philippsen (nphilipp) - * Adam Saleh (asaleh) + #fedora-admin, #fedora-noc, + admin@fedoraproject.org Servers:: log01, proxy0* Purpose:: Give interested parties information about the number of Fedora installations. Repositories:: - * https://pagure.io/mirrors-countme + * https://github.com/fedora-infra/mirrors-countme * https://pagure.io/fedora-infra/ansible/blob/main/f/roles/web-data-analysis == What it is @@ -51,10 +49,36 @@ log data) and b) shell scripts in the `web-data-analysis` role in Ansible: including proxies to `log01`. * `combineHttpLogs.sh` (Ansible) combines the logs for the different web sites which are scattered across the proxy hosts. -* `condense-mirrorlogs.sh` & `mirrorlist.py` (Ansible) extract hosts from the +* `condense-mirrorlogs.sh` & `mirrorlist.py` (Ansible) extracts hosts from the combined log data. * `countme-update.sh` (Ansible) drives `countme-update-rawdb.sh` & - `countme-update-totals.sh` (`mirrors-countme`) which generate statistics. + `countme-update-totals.sh` (`mirrors-countme`) which generates statistics. +* `countme-trim-raw` (`mirrors-countme`) to trim the intermediary database file + (``raw.db``). + +== Changes implemented in the Q2/2023 DNF mirrors-countme initiative + +* The “traditional“ statistics which were done before DNF + learned about the `countme` variable were reimplemented: Count any + individual IP sighted, no matter if with or without `countme`. This is + necessary to count systems which don’t have that feature in their DNF or + YUM, and – while giving different numbers – gives us an idea how things + develop when compared to the same numbers for more modern OSes. +* The ``countme-trim-raw`` tool was implemented, to trim the intermediary + database ``raw.db`` which contains necessary information gleamed from + parsing the merged log files. This database grows steadily and – with the + brought back counting of any individual IP sighted – quickly, so once these + data have been safely turned into the final statistics, we wanted a way to + remove them so that the local volume were it is stored doesn’t fill up + completely. +* The project repository was cleaned up, i.e. large data files used in + integration tests were removed because they made cloning the repository + unnecessarily slow, for a couple hundred KB of code, the repo was more than + 300 MB in size. In the context, the repository was moved from Pagure to + GitHub. +* Unused code was removed, the remaining code was refactored and condensed + to remove redundancies and comprehensive unit tests were added so that the + barrier to contributing is lower and changes are less risky. == Changes implemented in the Q3/2021 DNF Counting Initiative @@ -137,7 +161,7 @@ scripts and re-running them. == Where are the docs? -Here :) and at https://pagure.io/mirrors-countme/blob/main/f/README.md +Here :) and at https://github.com/fedora-infra/mirrors-countme == Is there data that needs to be backed up? @@ -160,30 +184,31 @@ branch of `mirrors-countme`. + From a clone of the upstream repository: + -.. In `countme/version.py`, bump `__version__` (e.g. to `0.0.7`) and +.. In `pyproject.toml`, bump `tool.poetry.version` (e.g. to `0.1.2`) and commit the change, e.g.: + .... -git commit -s -m "Bump version to 0.0.7" -- countme/version.py +git commit -s -m "Version 0.1.2" -- pyproject.toml .... .. Tag the previous change with a GPG-signed tag: + .... -git tag -s 0.0.7 +git tag -s 0.1.2 .... .. Push both the change and the tag: + .... -git push origin main 0.0.7 +git push origin main 0.1.2 .... .. Create a source tarball (this will be created as e.g. -`dist/mirrors-countme-0.0.7.tar.gz`): +`dist/mirrors_countme-0.1.2.tar.gz`): + .... -python setup.py sdist +poetry build .... -.. https://pagure.io/mirrors-countme/upload[Upload this tarball] to the -`mirrors-countme` releases folder +From the https://github.com/fedora-infra/mirrors-countme/tags[list of tags], +select “Create release” in the menu for the respective tag, and attach the +created tarball and wheel files to the created release. . Update and Build the `python-mirrors-countme` Fedora Package + From a clone of the Fedora package repository, in the `rawhide` branch: @@ -203,12 +228,12 @@ rpmspectool get python-mirrors-countme.spec .. Upload the source tarball to the lookaside cache: + .... -fedpkg new-sources mirrors-countme-0.0.7.tar.gz +fedpkg new-sources mirrors_countme-0.1.2.tar.gz .... .. Commit the changes to the repository, e.g.: + .... -git commit -s -m "Version 0.0.7" -- python-mirrors-countme.spec +git commit -s -m "Version 0.1.2" -- python-mirrors-countme.spec .... .. Push the changes and build: + @@ -244,7 +269,7 @@ koji tag-build epel8-infra-candidate more than a few minutes), e.g.: + .... -koji buildinfo python-mirrors-countme-0.0.7-1.el8 +koji buildinfo python-mirrors-countme-0.1.2-1.el8 .... + The build must be tagged with the corresponding `*-infra-stg` tag. @@ -281,9 +306,10 @@ All of this runs on `log01.iad2.fedoraproject.org` and is deployed through the `web-data-analysis` role and the `groups/logserver.yml` playbook, respectively. -The `mirrors-countme` upstream project publishes source tarballs here: +The `mirrors-countme` upstream project publishes source tarballs to their +corresponding releases in the repository on GitHub: -https://releases.pagure.org/mirrors-countme/ +https://github.com/fedora-infra/mirrors-countme/releases These are packaged in Fedora as the `python-mirrors-countme` (SRPM) and `python3-mirrors-countme` (RPM) packages. @@ -303,7 +329,7 @@ The same as anything else that deals with log data. Report bugs with `mirrors-countme` at its upstream project: -https://pagure.io/mirrors-countme/new_issue +https://github.com/fedora-infra/mirrors-countme/issues/new Anything concerning the cron jobs or other scripts should probably go into our Infrastructure tracker: