From 6fc255a4116ba3cdd88831d31de8105b77d42212 Mon Sep 17 00:00:00 2001 From: Adam Saleh Date: Wed, 26 Jan 2022 12:50:43 +0100 Subject: [PATCH] Split the bodhi SOP to dedployment and releng one --- modules/sysadmin_guide/nav.adoc | 3 +- .../sysadmin_guide/pages/bodhi-deploy.adoc | 278 ++++++++++++++++++ modules/sysadmin_guide/pages/bodhi.adoc | 213 +------------- 3 files changed, 281 insertions(+), 213 deletions(-) create mode 100644 modules/sysadmin_guide/pages/bodhi-deploy.adoc diff --git a/modules/sysadmin_guide/nav.adoc b/modules/sysadmin_guide/nav.adoc index 1d10b49..d18c91f 100644 --- a/modules/sysadmin_guide/nav.adoc +++ b/modules/sysadmin_guide/nav.adoc @@ -10,7 +10,8 @@ ** xref:aws-access.adoc[Amazon Web Services Access - SOP] ** xref:bastion-hosts-info.adoc[Fedora Bastion Hosts - SOP] ** xref:blockerbugs.adoc[Blockerbugs Infrastructure - SOP] -** xref:bodhi.adoc[Bodhi Infrastructure - SOP] +** xref:bodhi.adoc[Bodhi Infrastructure - Releng SOP] +** xref:bodhi-deploy.adoc[Bodhi Infrastructure - Deployment SOP] ** xref:bugzilla.adoc[Bugzilla Sync Infrastructure - SOP] ** xref:bugzilla2fedmsg.adoc[bugzilla2fedmsg - SOP] ** xref:collectd.adoc[Collectd - SOP] diff --git a/modules/sysadmin_guide/pages/bodhi-deploy.adoc b/modules/sysadmin_guide/pages/bodhi-deploy.adoc new file mode 100644 index 0000000..acf4956 --- /dev/null +++ b/modules/sysadmin_guide/pages/bodhi-deploy.adoc @@ -0,0 +1,278 @@ += Bodhi Infrastructure Release and Deployment SOP + +Bodhi is used by Fedora developers to submit potential package updates +for releases and to manage buildroot overrides. From here, bodhi handles +all of the dirty work, from sending around emails, dealing with Koji, to +composing the repositories. + +Bodhi production instance: https://bodhi.fedoraproject.org + +Bodhi project page: https://github.com/fedora-infra/bodhi + +=== Necessary permissions + +Groups to be member in + +* https://github.com/orgs/fedora-infra/teams/bodhi +* https://src.fedoraproject.org/group/infra-sig +* https://accounts.fedoraproject.org/group/infra-sig/ +* https://accounts.fedoraproject.org/group/packager/ +* https://accounts.fedoraproject.org/group/sysadmin/ +* https://accounts.fedoraproject.org/group/sysadmin-noc/ +* https://accounts.fedoraproject.org/group/sysadmin-bodhi/ + +In addition you should be named as app-owner in the bodhi os playbook. + +== Performing a bodhi upgrade + +=== Have the release in the official Bodhi repository ready + +Following steps were taken from +https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst_] +up to the point of building a beta-release.  + +Changing the specfile manually is tedious and error-prone, it is more +useful to just do several patch releases on staging, before we commit to +one to deploy on production. + +==== Releasing a patch-release + +Make sure you have all of the code you want to release in the branch. I.e. for 5.7.3 release, +you would be looking at https://github.com/fedora-infra/bodhi/tree/5.7 + +Easiest way to incorporate additional changes from PRs targetting develop, +is to tag them with `*-backports` tag, i.e. `5.7-backports`. Mergify will automatically create a cherry-picked PR +targetting the release branch. + +When you are ready to denote the new patch version: + +1. bump __version__ in ./bodhi/__init__.py +2. run the get-authors.py in ./news +3. run towncrier +4. review the docs/user/release_notes.rst +5. commit, push to your branch and make a PR against develop with the corect `*-backports` label + +After all of the tests successfully passed in the PRs, and both of them are merged, tag the release: +1. git checkout 5.7 +2. git pull origin 5.7 +3. git tag 5.7.3 +4. git push origin refs/tags/5.7.3 + +If you push the wrong tag, you can delete it by `git push origin :refs/tags/5.7.3` + +Beware, mergify creates the bacport PR only after the pr to develop was merged. + +==== Testing deployed Bodhi without making a release + +If the Bodhi changes don’t involve any of + +* Composer code +* Doesn’t include any upgrades to database with alembic + +You can opt to use the hotfix-style container-image that can be found +among the templates: +https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix_]   + +Change the env-vars in the docker-file-base-hotfix to reflect the commit +to be built. The image definition is not buildable locally, as it needs +the openshift environment to configure koji and kerberos. + +Change the buildconfig to point to the new definition: + +https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14_] + +Then commit and push the changes, to be able to deploy from batcave +with: + +sudo rbac-playbook openshift-apps/bodhi.yml -l staging + +Make sure to check os.stg.fedoraproject.org to see if the build completed successfully. + +==== Build Bodhi + +Bodhi is deployed from the infrastructure Koji repositories. At the time +of this writing, it is deployed from the f34-infra and f34-infra-stg +(for staging) repositories. Bodhi is built for these repositories from +the master branch of thehttps://src.fedoraproject.org/rpms/bodhi[ +]https://src.fedoraproject.org/rpms/bodhi[_bodhi dist-git repository_]. + +You can use fedpkg to download the package: +.... +fedpkg clone rpms/bodhi +.... + +Curently we are interested in f34 builds as all of the bodhi infra runs on f34 servers: + +.... +fedpkg switch-branch f34 +.... + +Edit the bodhi.spec specfile to have the correct release version. + +Download the sources to the SOURCES folder (if it doesn't exist, just mkdir -p /rpmbuild/SOURCES/) + +.... +wget https://github.com/fedora-infra/bodhi/archive/refs/tags/5.7.3.tar.gz -o /rpmbuild/SOURCES/bodhi-5.7.3.tar.gz +.... + +As an example, to build a Bodhi  for the f34-infra-stg repository, you +can use this command: + +.... +$ rpmbuild --define "dist .f34.infra" -bs bodhi.spec +.... + +Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm + +.... +$ koji build f34-infra ~/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm +.... + +When building a Bodhi release that is intended for production, we should +build from the production dist-git repo instead of uploading an SRPM: + +$ koji build f34-infra +git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37 + +All builds against the f34-infra build target will go into the +f34-infra-stg repository. If you wish to promote a build from staging to +production, you can do something like this command: + +$ koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.f34.infra + +=== Staging + +The upgrade playbook will apply configuration changes after running the +alembic upgrade. Sometimes you may need changes applied to the Bodhi +systems in order to get the upgrade playbook to succeed. If you are in +this situation, you can apply those changes by running the bodhi-backend +playbook: + +.... +sudo rbac-playbook -l staging groups/bodhi-backend.yml +.... + +In +the https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg, edit the bodhi_version setting to the version you wish to +deploy to staging. For example, to deploy version 5.7.3 to +staging, I would set that varible like this: + +bodhi_version:  ‘5.7.3’ + +The package itself will then be installed in container build by ‘dnf +install -y bodhi-server-\{\{bodhi_version}}’ + +Beware, when running the playbooks on the backend vm, it currently +installs the latest bodhi package available in the respective +infra/infra-stg tag. + +It can be useful to verify correct version is available on the backend, +i.e. for staging run + +.... +ssh bodhi-backend01.stg.iad2.fedoraproject.org +.... + +.... +dnf list bodhi-server --refresh --all --showduplicates +.... + +In future we will probably pin these in the VM as well. + +Make sure you have the correct docker-image set in the configuration. +in the ansible repo: +.... +vim roles/openshift-apps/bodhi/templates/buildconfig.yml +.... + +After you checked, and have the correct version, commit and create a PR for ansible repo. + +After it has been merged, run these commands: + +.... +# Synchronize the database from production to staging +$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging +# Upgrade the Bodhi backend on staging +$ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging +# Upgrade the Bodhi frontend on staging +$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging +.... + +==== Tests to run + +For compose test you can follow +https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta_] + +For automated health-check, you can run +https://pagure.io/fedora-ci/monitor-gating[_https://pagure.io/fedora-ci/monitor-gating_], + +It should be sufficient to check the openshift project +https://os.fedoraproject.org/console/project/monitor-gating[_https://os.fedoraproject.org/console/project/monitor-gating_] +where it is being run every few hours. There is a dashboard available +https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/6q2qpcvGz/monitor-gating + +==== Production + +After testing in staging, you should be able to move the rpm from +staging to production: + +.... +koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.fc34 +.... + +Even if the rpm version is pinned in your deploy, double check the +availability in the tag + +The upgrade playbook will apply configuration changes after running the +alembic upgrade. Sometimes you may need changes applied to the Bodhi +systems in order to get the upgrade playbook to succeed. If you are in +this situation, you can apply those changes by running the bodhi-backend +playbook: + +.... +sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend +.... + +In +thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[ +]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[_os_masters +inventory_], edit the bodhi_version setting to the version you wish to +deploy to production. For example, to deploy bodhi-5.7.3-0.f34.infra to +production, I would set that varible like this: + +.... +bodhi_version: "5.7.3" +.... + +To update the bodhi RPMs in production: + +.... +# Update the backend VMs (this will also run the migrations, if any) +$ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend +# Update the frontend +$ sudo rbac-playbook openshift-apps/bodhi.yml +.... + +All of the playbooks should finish with no failures. + +Afterwards, check +https://os.fedoraproject.org/console/project/bodhi/overview[_https://os.fedoraproject.org/console/project/bodhi/overview_] +to see if the image has been built and if it has been properly deployed. + +Things to watchout for + +* Correct rpm not being available for container build +** This is the reason to check the infra-tag for bodhi +* Failure to start the pod due to backend db mismatch +** Sometimes the alembic upgrade that is supposed to run as part of +playbooks/manual/upgrade/bodhi.yml fails silently +* Some issues resurface only after few days in production, resulting in +visible stack-traces in logs. + +=== Syncing the production database to staging + +This can be useful for testing issues with production data in staging: + +.... +$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging +.... diff --git a/modules/sysadmin_guide/pages/bodhi.adoc b/modules/sysadmin_guide/pages/bodhi.adoc index aa7a932..8adedcd 100644 --- a/modules/sysadmin_guide/pages/bodhi.adoc +++ b/modules/sysadmin_guide/pages/bodhi.adoc @@ -1,4 +1,4 @@ -= Bodhi Infrastructure SOP += Bodhi Infrastructure Releng SOP Bodhi is used by Fedora developers to submit potential package updates for releases and to manage buildroot overrides. From here, bodhi handles @@ -190,217 +190,6 @@ If a push fails for some reason, you can easily resume it on $ sudo -u apache bodhi-push --resume .... -== Performing a bodhi upgrade - -==== Have the release in the official Bodhi repository ready - -Follow steps in -https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst_] -up to the point of building a beta-release.  - -Changing the specfile manually is tedious and error-prone, it is more -useful to just do several patch releases on staging, before we commit to -one to deploy on production. - -==== Testing deployed Bodhi without making a release - -If the Bodhi changes don’t involve any of - -* Composer code -* Doesn’t include any upgrades to database with alembic - -You can opt to use the hotfix-style container-image that can be found -among the templates: -https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix_]   - -Change the env-vars in the docker-file-base-hotfix to reflect the commit -to be built. The image definition is not buildable locally, as it needs -the openshift environment to configure koji and kerberos. - -Change the buildconfig to point to the new definition: - -https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14_] - -I.e: - -[cols=",",] -|=== -| |\{% if env == "staging" %} -| |      \{\{ load_file('dockerfile-base-hotfix') | indent(8) }} -| |\{% else %} -| |      \{\{ load_file('dockerfile-base') | indent(8) }} -| |\{% endif %} -|=== - -Then commit and push the changes, to be able to deploy from batcave -with: - -sudo rbac-playbook openshift-apps/bodhi.yml -l staging - -==== Build Bodhi - -Bodhi is deployed from the infrastructure Koji repositories. At the time -of this writing, it is deployed from the f34-infra and f34-infra-stg -(for staging) repositories. Bodhi is built for these repositories from -the master branch of thehttps://src.fedoraproject.org/rpms/bodhi[ -]https://src.fedoraproject.org/rpms/bodhi[_bodhi dist-git repository_]. - -As an example, to build a Bodhi  for the f34-infra-stg repository, you -can use this command: - -.... -$ rpmbuild --define "dist .f34.infra" -bs bodhi.spec -.... - -Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm - -.... -$ koji build f34-infra -/home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm -.... - -When building a Bodhi release that is intended for production, we should -build from the production dist-git repo instead of uploading an SRPM: - -$ koji build f34-infra -git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37 - -All builds against the f34-infra build target will go into the -f34-infra-stg repository. If you wish to promote a build from staging to -production, you can do something like this command: - -$ koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.f34.infra - -=== Staging - -The upgrade playbook will apply configuration changes after running the -alembic upgrade. Sometimes you may need changes applied to the Bodhi -systems in order to get the upgrade playbook to succeed. If you are in -this situation, you can apply those changes by running the bodhi-backend -playbook: - -.... -sudo rbac-playbook -l staging groups/bodhi-backend.yml -.... - -In -thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[ -]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[_os_masters -inventory_], edit the bodhi_version setting to the version you wish to -deploy to staging. For example, to deploy version 5.7.3 to -staging, I would set that varible like this: - -bodhi_version:  ‘5.7.3’ - -The package itself will then be installed in container build by ‘dnf -install -y bodhi-server-\{\{bodhi_version}}’ - -Beware, when running the playbooks on the backend vm, it currently -installs the latest bodhi package available in the respective -infra/infra-stg tag. - -It can be useful to verify correct version is available on the backend, -i.e. for staging run - -.... -ssh bodhi-backend01.stg.iad2.fedoraproject.org -.... - -.... -dnf list bodhi-server --refresh --all --showduplicates -.... - -*MAYBE WE SHOULD PIN THIS AS WELL!!!* - -Run these commands: - -.... -# Synchronize the database from production to staging -$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging -# Upgrade the Bodhi backend on staging -$ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging -# Upgrade the Bodhi frontend on staging -$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging -.... - -==== Tests to run - -For compose test you can follow -https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta_] - -For automated health-check, you can run -https://pagure.io/fedora-ci/monitor-gating[_https://pagure.io/fedora-ci/monitor-gating_], - -It should be sufficient to check the openshift project -https://os.fedoraproject.org/console/project/monitor-gating[_https://os.fedoraproject.org/console/project/monitor-gating_] -where it is being run every few hours. There is a dashboard available -https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/6q2qpcvGz/monitor-gating - -==== Production - -After testing in staging, you should be able to move the rpm from -staging to production: - -.... -koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.fc34 -.... - -Even if the rpm version is pinned in your deploy, double check the -availability in the tag - -The upgrade playbook will apply configuration changes after running the -alembic upgrade. Sometimes you may need changes applied to the Bodhi -systems in order to get the upgrade playbook to succeed. If you are in -this situation, you can apply those changes by running the bodhi-backend -playbook: - -.... -sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend -.... - -In -thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[ -]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[_os_masters -inventory_], edit the bodhi_version setting to the version you wish to -deploy to production. For example, to deploy bodhi-5.7.3-0.f34.infra to -production, I would set that varible like this: - -.... -bodhi_version: "5.7.3" -.... - -To update the bodhi RPMs in production: - -.... -# Update the backend VMs (this will also run the migrations, if any) -$ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend -# Update the frontend -$ sudo rbac-playbook openshift-apps/bodhi.yml -.... - -All of the playbooks should finish with no failures. - -Afterwards, check -https://os.fedoraproject.org/console/project/bodhi/overview[_https://os.fedoraproject.org/console/project/bodhi/overview_] -to see if the image has been built and if it has been properly deployed. - -Things to watchout for - -* Correct rpm not being available for container build -** This is the reason to check the infra-tag for bodhi -* Failure to start the pod due to backend db mismatch -** Sometimes the alembic upgrade that is supposed to run as part of -playbooks/manual/upgrade/bodhi.yml fails silently -* Some issues resurface only after few days in production, resulting in -visible stack-traces in logs. - -=== Syncing the production database to staging - -This can be useful for testing issues with production data in staging: - -.... -$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging -.... == Release EOL