From 0158707f1ee233e3c5153ba6416219ee96299b05 Mon Sep 17 00:00:00 2001 From: Adam Saleh Date: Thu, 20 Jan 2022 11:31:07 +0100 Subject: [PATCH] Updating bodhi guide --- modules/sysadmin_guide/pages/bodhi.adoc | 168 +++++++++++++++++++----- 1 file changed, 137 insertions(+), 31 deletions(-) diff --git a/modules/sysadmin_guide/pages/bodhi.adoc b/modules/sysadmin_guide/pages/bodhi.adoc index c65fab2..aa7a932 100644 --- a/modules/sysadmin_guide/pages/bodhi.adoc +++ b/modules/sysadmin_guide/pages/bodhi.adoc @@ -192,37 +192,84 @@ $ sudo -u apache bodhi-push --resume == Performing a bodhi upgrade -=== Build Bodhi +==== Have the release in the official Bodhi repository ready + +Follow steps in +https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst_] +up to the point of building a beta-release.  + +Changing the specfile manually is tedious and error-prone, it is more +useful to just do several patch releases on staging, before we commit to +one to deploy on production. + +==== Testing deployed Bodhi without making a release + +If the Bodhi changes don’t involve any of + +* Composer code +* Doesn’t include any upgrades to database with alembic + +You can opt to use the hotfix-style container-image that can be found +among the templates: +https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix_]   + +Change the env-vars in the docker-file-base-hotfix to reflect the commit +to be built. The image definition is not buildable locally, as it needs +the openshift environment to configure koji and kerberos. + +Change the buildconfig to point to the new definition: + +https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14_] + +I.e: + +[cols=",",] +|=== +| |\{% if env == "staging" %} +| |      \{\{ load_file('dockerfile-base-hotfix') | indent(8) }} +| |\{% else %} +| |      \{\{ load_file('dockerfile-base') | indent(8) }} +| |\{% endif %} +|=== + +Then commit and push the changes, to be able to deploy from batcave +with: + +sudo rbac-playbook openshift-apps/bodhi.yml -l staging + +==== Build Bodhi Bodhi is deployed from the infrastructure Koji repositories. At the time -of this writing, it is deployed from the `f29-infra` and `f29-infra-stg` +of this writing, it is deployed from the f34-infra and f34-infra-stg (for staging) repositories. Bodhi is built for these repositories from -the `master` branch of the -https://src.fedoraproject.org/rpms/bodhi[bodhi dist-git repository]. +the master branch of thehttps://src.fedoraproject.org/rpms/bodhi[ +]https://src.fedoraproject.org/rpms/bodhi[_bodhi dist-git repository_]. -As an example, to build a Bodhi beta for the `f29-infra-stg` repository, -you can use this command: +As an example, to build a Bodhi  for the f34-infra-stg repository, you +can use this command: .... -$ rpmbuild --define "dist .fc29.infra" -bs bodhi.spec -Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-3.13.0-0.0.beta.e0ca5bc.fc29.infra.src.rpm -$ koji build f29-infra /home/bowlofeggs/rpmbuild/SRPMS/bodhi-3.13.0-0.0.beta.e0ca5bc.fc29.infra.src.rpm +$ rpmbuild --define "dist .f34.infra" -bs bodhi.spec +.... + +Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm + +.... +$ koji build f34-infra +/home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm .... When building a Bodhi release that is intended for production, we should build from the production dist-git repo instead of uploading an SRPM: -.... -$ koji build f29-infra git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37 -.... +$ koji build f34-infra +git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37 -All builds against the `f29-infra` build target will go into the -`f29-infra-stg` repository. If you wish to promote a build from staging -to production, you can do something like this command: +All builds against the f34-infra build target will go into the +f34-infra-stg repository. If you wish to promote a build from staging to +production, you can do something like this command: -.... -$ koji move-build f29-infra-stg f29-infra bodhi-3.13.0-1.fc29.infra -.... +$ koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.f34.infra === Staging @@ -236,16 +283,35 @@ playbook: sudo rbac-playbook -l staging groups/bodhi-backend.yml .... -In the -https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[os_masters inventory], -edit the `bodhi_version` setting to the version you wish to deploy to -staging. For example, to deploy `bodhi-3.13.0-1.fc29.infra` to staging, -I would set that varible like this: +In +thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[ +]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[_os_masters +inventory_], edit the bodhi_version setting to the version you wish to +deploy to staging. For example, to deploy version 5.7.3 to +staging, I would set that varible like this: + +bodhi_version:  ‘5.7.3’ + +The package itself will then be installed in container build by ‘dnf +install -y bodhi-server-\{\{bodhi_version}}’ + +Beware, when running the playbooks on the backend vm, it currently +installs the latest bodhi package available in the respective +infra/infra-stg tag. + +It can be useful to verify correct version is available on the backend, +i.e. for staging run .... -bodhi_version: "bodhi-3.13.0-1.fc29.infra" +ssh bodhi-backend01.stg.iad2.fedoraproject.org .... +.... +dnf list bodhi-server --refresh --all --showduplicates +.... + +*MAYBE WE SHOULD PIN THIS AS WELL!!!* + Run these commands: .... @@ -257,7 +323,30 @@ $ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging $ sudo rbac-playbook openshift-apps/bodhi.yml -l staging .... -=== Production +==== Tests to run + +For compose test you can follow +https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta_] + +For automated health-check, you can run +https://pagure.io/fedora-ci/monitor-gating[_https://pagure.io/fedora-ci/monitor-gating_], + +It should be sufficient to check the openshift project +https://os.fedoraproject.org/console/project/monitor-gating[_https://os.fedoraproject.org/console/project/monitor-gating_] +where it is being run every few hours. There is a dashboard available +https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/6q2qpcvGz/monitor-gating + +==== Production + +After testing in staging, you should be able to move the rpm from +staging to production: + +.... +koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.fc34 +.... + +Even if the rpm version is pinned in your deploy, double check the +availability in the tag The upgrade playbook will apply configuration changes after running the alembic upgrade. Sometimes you may need changes applied to the Bodhi @@ -269,14 +358,15 @@ playbook: sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend .... -In the -https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[os_masters inventory], -edit the `bodhi_version` setting to the version you wish to deploy to -production. For example, to deploy `bodhi-3.13.0-1.fc29.infra` to +In +thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[ +]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[_os_masters +inventory_], edit the bodhi_version setting to the version you wish to +deploy to production. For example, to deploy bodhi-5.7.3-0.f34.infra to production, I would set that varible like this: .... -bodhi_version: "bodhi-3.13.0-1.fc29.infra" +bodhi_version: "5.7.3" .... To update the bodhi RPMs in production: @@ -288,7 +378,23 @@ $ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend $ sudo rbac-playbook openshift-apps/bodhi.yml .... -== Syncing the production database to staging +All of the playbooks should finish with no failures. + +Afterwards, check +https://os.fedoraproject.org/console/project/bodhi/overview[_https://os.fedoraproject.org/console/project/bodhi/overview_] +to see if the image has been built and if it has been properly deployed. + +Things to watchout for + +* Correct rpm not being available for container build +** This is the reason to check the infra-tag for bodhi +* Failure to start the pod due to backend db mismatch +** Sometimes the alembic upgrade that is supposed to run as part of +playbooks/manual/upgrade/bodhi.yml fails silently +* Some issues resurface only after few days in production, resulting in +visible stack-traces in logs. + +=== Syncing the production database to staging This can be useful for testing issues with production data in staging: