2021-07-26 10:39:47 +02:00
= Bodhi Infrastructure SOP
Bodhi is used by Fedora developers to submit potential package updates
for releases and to manage buildroot overrides. From here, bodhi handles
all of the dirty work, from sending around emails, dealing with Koji, to
composing the repositories.
2021-08-18 11:44:52 +02:00
Bodhi production instance: https://bodhi.fedoraproject.org
Bodhi project page: https://github.com/fedora-infra/bodhi
2021-07-26 10:39:47 +02:00
== Contents
2021-08-18 11:44:52 +02:00
* <<_contact_information>>
* <<_adding_a_new_pending_release>>
* <<_pre_beta_bodhi_config>>
* <<_post_beta_bodhi_config>>
* <<_0_day_release_actions>>
* <<_configuring_all_bodhi_nodes>>
* <<_pushing_updates>>
* <<_monitoring_the_bodhi_composer_output>>
* <<_resuming_a_failed_push>>
* <<_performing_a_bodhi_upgrade>>
* <<_syncing_the_production_database_to_staging>>
* <<_release_eol>>
* <<_adding_notices_to_the_front_page_or_new_update_form>>
* <<_using_the_bodhi_shell_to_modify_updates_by_hand>>
* <<_using_the_bodhi_shell_to_fix_uniqueness_problems_with_e_mail_addresses>>
* <<_troubleshooting_and_resolution>>
2021-07-26 10:39:47 +02:00
== Contact Information
Owner::
Fedora Infrastructure Team
Contact::
#fedora-admin
Persons::
bowlofeggs
Location::
2021-08-18 11:44:52 +02:00
iad2
2021-07-26 10:39:47 +02:00
Servers::
2021-08-18 11:44:52 +02:00
* bodhi-backend01.iad2.fedoraproject.org (composer)
2021-07-26 10:39:47 +02:00
* os.fedoraproject.org (web front end and backend task workers for
non-compose tasks)
2021-08-18 11:44:52 +02:00
* bodhi-backend01.stg.iad2.fedoraproject.org (staging composer)
2021-07-26 10:39:47 +02:00
* os.stg.fedoraproject.org (staging web front end and backend task
workers for non-compose tasks)
Purpose::
Push package updates, and handle new submissions.
== Adding a new pending release
Adding and modifying releases is done using the
2021-08-18 11:44:52 +02:00
_bodhi-manage-releases_ tool.
2021-07-26 10:39:47 +02:00
You can add a new pending release by running this command:
....
2021-10-24 08:42:08 +00:00
bodhi-manage-releases create --name F23 --long-name "Fedora 23" --id-prefix FEDORA --version 23 --branch rawhide --dist-tag f23 --stable-tag f23-updates --testing-tag f23-updates-testing --candidate-tag f23-updates-candidate --pending-stable-tag f23-updates-pending --pending-testing-tag f23-updates-testing-pending --override-tag f23-override --package-manager dnf --testing-repository updates-testing --state pending --not-composed-by-bodhi --create-automatic-updates
2021-07-26 10:39:47 +02:00
....
== Pre-Beta Bodhi config
2021-10-24 08:42:08 +00:00
Disable automatic updates creation and enable composes by Bodhi:
....
bodhi-manage-releases edit --name F23 --composed-by-bodhi --no-create-automatic-updates --branch f23
....
2021-07-26 10:39:47 +02:00
Enable pre_beta policy in bodhi config in ansible.::::
ansible/roles/bodhi2/base/templates/production.ini.j2
Uncomment or add the following lines:
....
#f29.status = pre_beta
#f29.pre_beta.mandatory_days_in_testing = 3
#f29.pre_beta.critpath.min_karma = 1
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
....
== Post-Beta Bodhi config
Enable post_beta policy in bodhi config in ansible.::::
ansible/roles/bodhi2/base/templates/production.ini.j2
Comment or remove the following lines corresponding to pre_beta policy:
....
#f29.status = pre_beta
#f29.pre_beta.mandatory_days_in_testing = 3
#f29.pre_beta.critpath.min_karma = 1
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
....
Uncomment or add the following lines for post_beta policy
....
#f29.status = post_beta
#f29.post_beta.mandatory_days_in_testing = 7
#f29.post_beta.critpath.min_karma = 2
#f29.post_beta.critpath.stable_after_days_without_negative_karma = 14
....
== 0-day Release Actions
* update atomic config
* run the ansible playbook
Going from pending to a proper release in bodhi requires a few steps:
Change state from pending to current:
....
bodhi-manage-releases edit --name F23 --state current
....
You may also need to disable any pre-beta or post-beta policy defined in
the bodhi config in ansible.:
....
ansible/roles/bodhi2/base/templates/production.ini.j2
....
Uncomment or remove the lines related to pre and post beta polcy
....
#f29.status = post_beta
#f29.post_beta.mandatory_days_in_testing = 7
#f29.post_beta.critpath.min_karma = 2
#f29.post_beta.critpath.stable_after_days_without_negative_karma = 14
#f29.status = pre_beta
#f29.pre_beta.mandatory_days_in_testing = 3
#f29.pre_beta.critpath.min_karma = 1
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
....
== Configuring all bodhi nodes
2021-08-18 11:44:52 +02:00
Run this command from the _ansible_ checkout to configure
2021-07-26 10:39:47 +02:00
all of bodhi in production:
....
# This will configure the backends
$ sudo rbac-playbook playbooks/groups/bodhi2.yml
# This will configure the frontend
$ sudo rbac-playbook openshift-apps/bodhi.yml
....
== Pushing updates
2021-08-18 11:44:52 +02:00
SSH into the `bodhi-backend01` machine and run:
2021-07-26 10:39:47 +02:00
....
$ sudo -u apache bodhi-push
....
You can restrict the updates by release and/or request:
....
$ sudo -u apache bodhi-push --releases f23,f22 --request stable
....
You can also push specific builds:
....
$ sudo -u apache bodhi-push --builds openssl-1.0.1k-14.fc22,openssl-1.0.1k-14.fc23
....
This will display a list of updates that are ready to be pushed.
== Monitoring the bodhi composer output
You can monitor the bodhi composer via the `bodhi` CLI tool, or via the
systemd journal on `bodhi-backend01`:
....
# From the comfort of your own laptop.
$ bodhi composes list
# From bodhi-backend01
$ journalctl -f -u fedmsg-hub
....
== Resuming a failed push
If a push fails for some reason, you can easily resume it on
`bodhi-backend01` by running:
....
$ sudo -u apache bodhi-push --resume
....
== Performing a bodhi upgrade
2022-01-20 11:31:07 +01:00
==== Have the release in the official Bodhi repository ready
Follow steps in
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst_]
up to the point of building a beta-release.
Changing the specfile manually is tedious and error-prone, it is more
useful to just do several patch releases on staging, before we commit to
one to deploy on production.
==== Testing deployed Bodhi without making a release
If the Bodhi changes don’
* Composer code
* Doesn’
You can opt to use the hotfix-style container-image that can be found
among the templates:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix_]
Change the env-vars in the docker-file-base-hotfix to reflect the commit
to be built. The image definition is not buildable locally, as it needs
the openshift environment to configure koji and kerberos.
Change the buildconfig to point to the new definition:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14_]
I.e:
[cols=",",]
|===
| |\{% if env == "staging" %}
| |
| |\{% else %}
| |
| |\{% endif %}
|===
Then commit and push the changes, to be able to deploy from batcave
with:
sudo rbac-playbook openshift-apps/bodhi.yml -l staging
==== Build Bodhi
2021-07-26 10:39:47 +02:00
Bodhi is deployed from the infrastructure Koji repositories. At the time
2022-01-20 11:31:07 +01:00
of this writing, it is deployed from the f34-infra and f34-infra-stg
2021-07-26 10:39:47 +02:00
(for staging) repositories. Bodhi is built for these repositories from
2022-01-20 11:31:07 +01:00
the master branch of thehttps://src.fedoraproject.org/rpms/bodhi[
]https://src.fedoraproject.org/rpms/bodhi[_bodhi dist-git repository_].
2021-07-26 10:39:47 +02:00
2022-01-20 11:31:07 +01:00
As an example, to build a Bodhi
can use this command:
2021-07-26 10:39:47 +02:00
....
2022-01-20 11:31:07 +01:00
$ rpmbuild --define "dist .f34.infra" -bs bodhi.spec
2021-07-26 10:39:47 +02:00
....
2022-01-20 11:31:07 +01:00
Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
2021-07-26 10:39:47 +02:00
....
2022-01-20 11:31:07 +01:00
$ koji build f34-infra
/home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
2021-07-26 10:39:47 +02:00
....
2022-01-20 11:31:07 +01:00
When building a Bodhi release that is intended for production, we should
build from the production dist-git repo instead of uploading an SRPM:
2021-07-26 10:39:47 +02:00
2022-01-20 11:31:07 +01:00
$ koji build f34-infra
git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37
All builds against the f34-infra build target will go into the
f34-infra-stg repository. If you wish to promote a build from staging to
production, you can do something like this command:
$ koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.f34.infra
2021-07-26 10:39:47 +02:00
=== Staging
The upgrade playbook will apply configuration changes after running the
alembic upgrade. Sometimes you may need changes applied to the Bodhi
systems in order to get the upgrade playbook to succeed. If you are in
this situation, you can apply those changes by running the bodhi-backend
playbook:
....
sudo rbac-playbook -l staging groups/bodhi-backend.yml
....
2022-01-20 11:31:07 +01:00
In
thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[
]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[_os_masters
inventory_], edit the bodhi_version setting to the version you wish to
deploy to staging. For example, to deploy version 5.7.3 to
staging, I would set that varible like this:
bodhi_version: ‘ ’
The package itself will then be installed in container build by ‘
install -y bodhi-server-\{\{bodhi_version}}’
Beware, when running the playbooks on the backend vm, it currently
installs the latest bodhi package available in the respective
infra/infra-stg tag.
It can be useful to verify correct version is available on the backend,
i.e. for staging run
2021-07-26 10:39:47 +02:00
....
2022-01-20 11:31:07 +01:00
ssh bodhi-backend01.stg.iad2.fedoraproject.org
2021-07-26 10:39:47 +02:00
....
2022-01-20 11:31:07 +01:00
....
dnf list bodhi-server --refresh --all --showduplicates
....
*MAYBE WE SHOULD PIN THIS AS WELL!!!*
2021-07-26 10:39:47 +02:00
Run these commands:
....
# Synchronize the database from production to staging
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
# Upgrade the Bodhi backend on staging
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging
# Upgrade the Bodhi frontend on staging
$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging
....
2022-01-20 11:31:07 +01:00
==== Tests to run
For compose test you can follow
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta_]
For automated health-check, you can run
https://pagure.io/fedora-ci/monitor-gating[_https://pagure.io/fedora-ci/monitor-gating_],
It should be sufficient to check the openshift project
https://os.fedoraproject.org/console/project/monitor-gating[_https://os.fedoraproject.org/console/project/monitor-gating_]
where it is being run every few hours. There is a dashboard available
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/6q2qpcvGz/monitor-gating
==== Production
After testing in staging, you should be able to move the rpm from
staging to production:
....
koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.fc34
....
Even if the rpm version is pinned in your deploy, double check the
availability in the tag
2021-07-26 10:39:47 +02:00
The upgrade playbook will apply configuration changes after running the
alembic upgrade. Sometimes you may need changes applied to the Bodhi
systems in order to get the upgrade playbook to succeed. If you are in
this situation, you can apply those changes by running the bodhi-backend
playbook:
....
sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend
....
2022-01-20 11:31:07 +01:00
In
thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[
]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[_os_masters
inventory_], edit the bodhi_version setting to the version you wish to
deploy to production. For example, to deploy bodhi-5.7.3-0.f34.infra to
2021-07-26 10:39:47 +02:00
production, I would set that varible like this:
....
2022-01-20 11:31:07 +01:00
bodhi_version: "5.7.3"
2021-07-26 10:39:47 +02:00
....
To update the bodhi RPMs in production:
....
# Update the backend VMs (this will also run the migrations, if any)
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend
# Update the frontend
$ sudo rbac-playbook openshift-apps/bodhi.yml
....
2022-01-20 11:31:07 +01:00
All of the playbooks should finish with no failures.
Afterwards, check
https://os.fedoraproject.org/console/project/bodhi/overview[_https://os.fedoraproject.org/console/project/bodhi/overview_]
to see if the image has been built and if it has been properly deployed.
Things to watchout for
* Correct rpm not being available for container build
** This is the reason to check the infra-tag for bodhi
* Failure to start the pod due to backend db mismatch
** Sometimes the alembic upgrade that is supposed to run as part of
playbooks/manual/upgrade/bodhi.yml fails silently
* Some issues resurface only after few days in production, resulting in
visible stack-traces in logs.
=== Syncing the production database to staging
2021-07-26 10:39:47 +02:00
This can be useful for testing issues with production data in staging:
....
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
....
== Release EOL
....
bodhi-manage-releases edit --name F21 --state archived
....
== Adding notices to the front page or new update form
You can easily add notification messages to the front page of bodhi
2021-08-18 11:44:52 +02:00
using the _frontpage_notice_ option in
_ansible/roles/bodhi2/base/templates/production.ini.j2_. If
2021-07-26 10:39:47 +02:00
you want to flash a message on the New Update Form, you can use the
2021-08-18 11:44:52 +02:00
_newupdate_notice_ variable instead. This can be useful for
2021-07-26 10:39:47 +02:00
announcing things like service outages, etc.
== Using the Bodhi Shell to modify updates by hand
The "bodhi shell" is a Python shell with the SQLAlchemy session and
transaction manager initialized. It can be run from any
production/staging backend instance and allows you to modify any models
by hand.
....
sudo pshell /etc/bodhi/production.ini
# Execute a script that sets up the `db` and provides a `delete_update` function.
# This will eventually be shipped in the bodhi package, but can also be found here.
# https://raw.githubusercontent.com/fedora-infra/bodhi/develop/tools/shelldb.py
>>> execfile('shelldb.py')
....
2021-08-18 11:44:52 +02:00
At this point you have access to a _db_ SQLAlchemy Session
instance, a _t_ _transaction module_, and
_m_ for the _bodhi.models_.
2021-07-26 10:39:47 +02:00
....
# Fetch an update, and tweak it as necessary.
>>> up = m.Update.get(u'u'FEDORA-2016-4d226a5f7e', db)
# Commit the transaction
>>> t.commit()
....
Here is an example of merging two updates together and deleting the
original.
....
>>> up = m.Update.get(u'FEDORA-2016-4d226a5f7e', db)
>>> up.builds
[<Build {'epoch': 0, 'nvr': u'resteasy-3.0.17-2.fc24'}>, <Build {'epoch': 0, 'nvr': u'pki-core-10.3.5-1.fc24'}>]
>>> b = up.builds[0]
>>> up2 = m.Update.get(u'FEDORA-2016-5f63a874ca', db)
>>> up2.builds
[<Build {'epoch': 0, 'nvr': u'resteasy-3.0.17-3.fc24'}>]
>>> up.builds.remove(b)
>>> up.builds.append(up2.builds[0])
>>> delete_update(up2)
>>> t.commit()
....
== Using the Bodhi shell to fix uniqueness problems with e-mail addresses
Bodhi currently enforces uniqueness on user e-mail addresses. There is
https://github.com/fedora-infra/bodhi/issues/2387[an issue] filed to
drop this upstream, but for the time being the constraint is enforced.
This can be a problem for users who have more than one FAS account if
they make one account use an e-mail address that was previously used by
another account, if that other account has not logged into Bodhi since
it was changed to use a different address. One way the user can fix this
themselves is to log in to Bodhi with the old account so that Bodhi
learns about its new address. However, an admin can also fix this by
hand by using the Bodhi shell.
For example, suppose a user has created `user_1` and `user_2`. Suppose
that `user_1` used to use `email_a@example.com` but has been changed to
use `email_b@example.com` in FAS, and `user_2` is now configured to use
`email_a@example.com` in FAS. If `user_2` attempts to log in to Bodhi,
it will cause a uniqueness violation since Bodhi does not know that
`user_1` has changed to `email_b@example.com`. The user can simply log
in as `user_1` to fix this, which will cause Bodhi to update its e-mail
address to `email_b@example.com`. Or an admin can fix it with a shell on
one of the Bodhi backend servers like this:
....
[bowlofeggs@bodhi-backend02 ~][PROD]$ sudo -u apache pshell /etc/bodhi/production.ini
2018-05-29 20:21:36,366 INFO [bodhi][MainThread] Using python-bugzilla
2018-05-29 20:21:36,367 DEBUG [bodhi][MainThread] Using Koji Buildsystem
2018-05-29 20:21:42,559 INFO [bodhi.server][MainThread] Bodhi ready and at your service!
Python 2.7.14 (default, Mar 14 2018, 13:36:31)
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
Type "help" for more information.
Environment:
app The WSGI application.
registry Active Pyramid registry.
request Active request object.
root Root of the default resource tree.
root_factory Default root factory used to create `root`.
Custom Variables:
m bodhi.server.models
>>> u = m.User.query.filter_by(name=u'user_1').one()
>>> u.email = u'email_b@example.com'
>>> m.Session().commit()
....
== Troubleshooting and Resolution
=== Atomic OSTree compose failure
2021-08-18 11:44:52 +02:00
If the Atomic OSTree compose fails with some sort of
_Device or Resource busy_ error, then run _mount_ to see if there
are any stray _tmpfs_ mounts still active:
2021-07-26 10:39:47 +02:00
....
tmpfs on /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq type tmpfs (rw,relatime,seclabel,mode=755)
....
2021-08-18 11:44:52 +02:00
You can then
_umount /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq_
2021-07-26 10:39:47 +02:00
and resume the push again.
=== nfs repodata cache IOError
Sometimes you may hit an IOError during the updateinfo.xml generation
process from createrepo_c:
....
IOError: Cannot open /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml: File /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml doesn't exists or not a regular file
....
This issue will be resolved with NFSv4, but in the mean time it can be
2021-08-18 11:44:52 +02:00
worked around by removing the _.repocache_ directory and
2021-07-26 10:39:47 +02:00
resuming the push:
....
rm -fr /mnt/koji/mash/updates/epel7.repocache
....
