= Bodhi Infrastructure Release and Deployment SOP

Bodhi is used by Fedora developers to submit potential package updates
for releases and to manage buildroot overrides. From here, bodhi handles
all of the dirty work, from sending around emails, dealing with Koji, to
composing the repositories.

Bodhi production instance: https://bodhi.fedoraproject.org

Bodhi project page: https://github.com/fedora-infra/bodhi

See also: xref:bodhi.adoc[Bodhi Infrastructure - Releng SOP]

== Performing a bodhi upgrade

=== Necessary permissions

Groups to be member in:

* https://github.com/orgs/fedora-infra/teams/bodhi
* https://src.fedoraproject.org/group/infra-sig
* https://accounts.fedoraproject.org/group/infra-sig/
* https://accounts.fedoraproject.org/group/packager/
* https://accounts.fedoraproject.org/group/sysadmin/
* https://accounts.fedoraproject.org/group/sysadmin-noc/
* https://accounts.fedoraproject.org/group/sysadmin-bodhi/

In addition you should be named as app-owner in the bodhi os playbook.

=== Have the release in the official Bodhi repository ready

Following steps were taken from
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst_]
up to the point of building a beta-release. 

Changing the specfile manually is tedious and error-prone, it is more
useful to just do several patch releases on staging, before we commit to
one to deploy on production.

==== Releasing a patch-release

Make sure you have all of the code you want to release in the branch. I.e. for 5.7.3 release,
you would be looking at https://github.com/fedora-infra/bodhi/tree/5.7

Easiest way to incorporate additional changes from PRs targetting develop,
is to tag them with `*-backports` tag, i.e. `5.7-backports`. Mergify will automatically create a cherry-picked PR
targetting the release branch.

When you are ready to denote the new patch version:

1. bump __version__ in ./bodhi/__init__.py
2. run the get-authors.py in ./news
3. run towncrier
4. review the docs/user/release_notes.rst
5. commit, push to your branch and make a PR against develop with the corect `*-backports` label 

After all of the tests successfully passed in the PRs, and both of them are merged, tag the release:
1. git checkout 5.7
2. git pull origin 5.7
3. git tag 5.7.3
4. git push origin refs/tags/5.7.3

If you push the wrong tag, you can delete it by `git push origin :refs/tags/5.7.3`

Beware, mergify creates the bacport PR only after the pr to develop was merged.

==== Testing deployed Bodhi without making a release

If the Bodhi changes don’t involve any of

* Composer code
* Doesn’t include any upgrades to database with alembic

You can opt to use the hotfix-style container-image that can be found
among the templates:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix_]  

Change the env-vars in the docker-file-base-hotfix to reflect the commit
to be built. The image definition is not buildable locally, as it needs
the openshift environment to configure koji and kerberos.

Change the buildconfig to point to the new definition:

https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14_]

Then commit and push the changes, to be able to deploy from batcave
with:

sudo rbac-playbook openshift-apps/bodhi.yml -l staging

Make sure to check os.stg.fedoraproject.org to see if the build completed successfully.

==== Build Bodhi

Bodhi is deployed from the infrastructure Koji repositories. At the time
of this writing, it is deployed from the f34-infra and f34-infra-stg
(for staging) repositories. Bodhi is built for these repositories from
the master branch of thehttps://src.fedoraproject.org/rpms/bodhi[
]https://src.fedoraproject.org/rpms/bodhi[_bodhi dist-git repository_].

You can use fedpkg to download the package:
....
fedpkg clone rpms/bodhi
....

Curently we are interested in f34 builds as all of the bodhi infra runs on f34 servers:

....
fedpkg switch-branch f34
....

Edit the bodhi.spec specfile to have the correct release version.

Download the sources to the SOURCES folder (if it doesn't exist, just mkdir -p /rpmbuild/SOURCES/)

....
wget https://github.com/fedora-infra/bodhi/archive/refs/tags/5.7.3.tar.gz -o /rpmbuild/SOURCES/bodhi-5.7.3.tar.gz
....

As an example, to build a Bodhi  for the f34-infra-stg repository, you
can use this command:

....
$ rpmbuild --define "dist .f34.infra" -bs bodhi.spec
....

Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm

....
$ koji build f34-infra ~/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
....

When building a Bodhi release that is intended for production, we should
build from the production dist-git repo instead of uploading an SRPM:

$ koji build f34-infra
git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37

All builds against the f34-infra build target will go into the
f34-infra-stg repository. If you wish to promote a build from staging to
production, you can do something like this command:

$ koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.f34.infra

=== Staging

The upgrade playbook will apply configuration changes after running the
alembic upgrade. Sometimes you may need changes applied to the Bodhi
systems in order to get the upgrade playbook to succeed. If you are in
this situation, you can apply those changes by running the bodhi-backend
playbook:

....
sudo rbac-playbook -l staging groups/bodhi-backend.yml
....

In
the https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg, edit the bodhi_version setting to the version you wish to
deploy to staging. For example, to deploy version 5.7.3 to
staging, I would set that varible like this:

bodhi_version:  ‘5.7.3’

The package itself will then be installed in container build by ‘dnf
install -y bodhi-server-\{\{bodhi_version}}’

Beware, when running the playbooks on the backend vm, it currently
installs the latest bodhi package available in the respective
infra/infra-stg tag.

It can be useful to verify correct version is available on the backend,
i.e. for staging run

....
ssh bodhi-backend01.stg.iad2.fedoraproject.org
....

....
dnf list bodhi-server --refresh --all --showduplicates
....

In future we will probably pin these in the VM as well.

Make sure you have the correct docker-image set in the configuration.
in the ansible repo:
....
vim roles/openshift-apps/bodhi/templates/buildconfig.yml
....

After you checked, and have the correct version, commit and create a PR for ansible repo.

After it has been merged, run these commands:

....
# Synchronize the database from production to staging
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
# Upgrade the Bodhi backend on staging
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging
# Upgrade the Bodhi frontend on staging
$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging
....

==== Tests to run

For compose test you can follow
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta_]

For automated health-check, you can run
https://pagure.io/fedora-ci/monitor-gating[_https://pagure.io/fedora-ci/monitor-gating_],

It should be sufficient to check the openshift project
https://os.fedoraproject.org/console/project/monitor-gating[_https://os.fedoraproject.org/console/project/monitor-gating_]
where it is being run every few hours. There is a dashboard available
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/6q2qpcvGz/monitor-gating

==== Production

After testing in staging, you should be able to move the rpm from
staging to production:

....
koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.fc34
....

Even if the rpm version is pinned in your deploy, double check the
availability in the tag

The upgrade playbook will apply configuration changes after running the
alembic upgrade. Sometimes you may need changes applied to the Bodhi
systems in order to get the upgrade playbook to succeed. If you are in
this situation, you can apply those changes by running the bodhi-backend
playbook:

....
sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend
....

In
thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[
]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[_os_masters
inventory_], edit the bodhi_version setting to the version you wish to
deploy to production. For example, to deploy bodhi-5.7.3-0.f34.infra to
production, I would set that varible like this:

....
bodhi_version: "5.7.3"
....

To update the bodhi RPMs in production:

....
# Update the backend VMs (this will also run the migrations, if any)
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend
# Update the frontend
$ sudo rbac-playbook openshift-apps/bodhi.yml
....

All of the playbooks should finish with no failures.

Afterwards, check
https://os.fedoraproject.org/console/project/bodhi/overview[_https://os.fedoraproject.org/console/project/bodhi/overview_]
to see if the image has been built and if it has been properly deployed.

Things to watchout for

* Correct rpm not being available for container build
** This is the reason to check the infra-tag for bodhi
* Failure to start the pod due to backend db mismatch
** Sometimes the alembic upgrade that is supposed to run as part of
playbooks/manual/upgrade/bodhi.yml fails silently
* Some issues resurface only after few days in production, resulting in
visible stack-traces in logs.

=== Syncing the production database to staging

This can be useful for testing issues with production data in staging:

....
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
....
