431 lines
13 KiB
Text
431 lines
13 KiB
Text
= Bodhi Infrastructure SOP
|
|
|
|
Bodhi is used by Fedora developers to submit potential package updates
|
|
for releases and to manage buildroot overrides. From here, bodhi handles
|
|
all of the dirty work, from sending around emails, dealing with Koji, to
|
|
composing the repositories.
|
|
|
|
Bodhi production instance: https://bodhi.fedoraproject.org
|
|
|
|
Bodhi project page: https://github.com/fedora-infra/bodhi
|
|
|
|
== Contents
|
|
|
|
* <<_contact_information>>
|
|
* <<_adding_a_new_pending_release>>
|
|
* <<_pre_beta_bodhi_config>>
|
|
* <<_post_beta_bodhi_config>>
|
|
* <<_0_day_release_actions>>
|
|
* <<_configuring_all_bodhi_nodes>>
|
|
* <<_pushing_updates>>
|
|
* <<_monitoring_the_bodhi_composer_output>>
|
|
* <<_resuming_a_failed_push>>
|
|
* <<_performing_a_bodhi_upgrade>>
|
|
* <<_syncing_the_production_database_to_staging>>
|
|
* <<_release_eol>>
|
|
* <<_adding_notices_to_the_front_page_or_new_update_form>>
|
|
* <<_using_the_bodhi_shell_to_modify_updates_by_hand>>
|
|
* <<_using_the_bodhi_shell_to_fix_uniqueness_problems_with_e_mail_addresses>>
|
|
* <<_troubleshooting_and_resolution>>
|
|
|
|
== Contact Information
|
|
|
|
Owner::
|
|
Fedora Infrastructure Team
|
|
Contact::
|
|
#fedora-admin
|
|
Persons::
|
|
bowlofeggs
|
|
Location::
|
|
iad2
|
|
Servers::
|
|
* bodhi-backend01.iad2.fedoraproject.org (composer)
|
|
* os.fedoraproject.org (web front end and backend task workers for
|
|
non-compose tasks)
|
|
* bodhi-backend01.stg.iad2.fedoraproject.org (staging composer)
|
|
* os.stg.fedoraproject.org (staging web front end and backend task
|
|
workers for non-compose tasks)
|
|
Purpose::
|
|
Push package updates, and handle new submissions.
|
|
|
|
== Adding a new pending release
|
|
|
|
Adding and modifying releases is done using the
|
|
_bodhi-manage-releases_ tool.
|
|
|
|
You can add a new pending release by running this command:
|
|
|
|
....
|
|
bodhi-manage-releases create --name F23 --long-name "Fedora 23" --id-prefix FEDORA --version 23 --branch f23 --dist-tag f23 --stable-tag f23-updates --testing-tag f23-updates-testing --candidate-tag f23-updates-candidate --pending-stable-tag f23-updates-pending --pending-testing-tag f23-updates-testing-pending --override-tag f23-override --state pending
|
|
....
|
|
|
|
== Pre-Beta Bodhi config
|
|
|
|
Enable pre_beta policy in bodhi config in ansible.::::
|
|
ansible/roles/bodhi2/base/templates/production.ini.j2
|
|
|
|
Uncomment or add the following lines:
|
|
|
|
....
|
|
#f29.status = pre_beta
|
|
#f29.pre_beta.mandatory_days_in_testing = 3
|
|
#f29.pre_beta.critpath.min_karma = 1
|
|
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
|
|
....
|
|
|
|
== Post-Beta Bodhi config
|
|
|
|
Enable post_beta policy in bodhi config in ansible.::::
|
|
ansible/roles/bodhi2/base/templates/production.ini.j2
|
|
|
|
Comment or remove the following lines corresponding to pre_beta policy:
|
|
|
|
....
|
|
#f29.status = pre_beta
|
|
#f29.pre_beta.mandatory_days_in_testing = 3
|
|
#f29.pre_beta.critpath.min_karma = 1
|
|
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
|
|
....
|
|
|
|
Uncomment or add the following lines for post_beta policy
|
|
|
|
....
|
|
#f29.status = post_beta
|
|
#f29.post_beta.mandatory_days_in_testing = 7
|
|
#f29.post_beta.critpath.min_karma = 2
|
|
#f29.post_beta.critpath.stable_after_days_without_negative_karma = 14
|
|
....
|
|
|
|
== 0-day Release Actions
|
|
|
|
* update atomic config
|
|
* run the ansible playbook
|
|
|
|
Going from pending to a proper release in bodhi requires a few steps:
|
|
|
|
Change state from pending to current:
|
|
|
|
....
|
|
bodhi-manage-releases edit --name F23 --state current
|
|
....
|
|
|
|
You may also need to disable any pre-beta or post-beta policy defined in
|
|
the bodhi config in ansible.:
|
|
|
|
....
|
|
ansible/roles/bodhi2/base/templates/production.ini.j2
|
|
....
|
|
|
|
Uncomment or remove the lines related to pre and post beta polcy
|
|
|
|
....
|
|
#f29.status = post_beta
|
|
#f29.post_beta.mandatory_days_in_testing = 7
|
|
#f29.post_beta.critpath.min_karma = 2
|
|
#f29.post_beta.critpath.stable_after_days_without_negative_karma = 14
|
|
#f29.status = pre_beta
|
|
#f29.pre_beta.mandatory_days_in_testing = 3
|
|
#f29.pre_beta.critpath.min_karma = 1
|
|
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
|
|
....
|
|
|
|
== Configuring all bodhi nodes
|
|
|
|
Run this command from the _ansible_ checkout to configure
|
|
all of bodhi in production:
|
|
|
|
....
|
|
# This will configure the backends
|
|
$ sudo rbac-playbook playbooks/groups/bodhi2.yml
|
|
# This will configure the frontend
|
|
$ sudo rbac-playbook openshift-apps/bodhi.yml
|
|
....
|
|
|
|
== Pushing updates
|
|
|
|
SSH into the `bodhi-backend01` machine and run:
|
|
|
|
....
|
|
$ sudo -u apache bodhi-push
|
|
....
|
|
|
|
You can restrict the updates by release and/or request:
|
|
|
|
....
|
|
$ sudo -u apache bodhi-push --releases f23,f22 --request stable
|
|
....
|
|
|
|
You can also push specific builds:
|
|
|
|
....
|
|
$ sudo -u apache bodhi-push --builds openssl-1.0.1k-14.fc22,openssl-1.0.1k-14.fc23
|
|
....
|
|
|
|
This will display a list of updates that are ready to be pushed.
|
|
|
|
== Monitoring the bodhi composer output
|
|
|
|
You can monitor the bodhi composer via the `bodhi` CLI tool, or via the
|
|
systemd journal on `bodhi-backend01`:
|
|
|
|
....
|
|
# From the comfort of your own laptop.
|
|
$ bodhi composes list
|
|
# From bodhi-backend01
|
|
$ journalctl -f -u fedmsg-hub
|
|
....
|
|
|
|
== Resuming a failed push
|
|
|
|
If a push fails for some reason, you can easily resume it on
|
|
`bodhi-backend01` by running:
|
|
|
|
....
|
|
$ sudo -u apache bodhi-push --resume
|
|
....
|
|
|
|
== Performing a bodhi upgrade
|
|
|
|
=== Build Bodhi
|
|
|
|
Bodhi is deployed from the infrastructure Koji repositories. At the time
|
|
of this writing, it is deployed from the `f29-infra` and `f29-infra-stg`
|
|
(for staging) repositories. Bodhi is built for these repositories from
|
|
the `master` branch of the
|
|
https://src.fedoraproject.org/rpms/bodhi[bodhi dist-git repository].
|
|
|
|
As an example, to build a Bodhi beta for the `f29-infra-stg` repository,
|
|
you can use this command:
|
|
|
|
....
|
|
$ rpmbuild --define "dist .fc29.infra" -bs bodhi.spec
|
|
Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-3.13.0-0.0.beta.e0ca5bc.fc29.infra.src.rpm
|
|
$ koji build f29-infra /home/bowlofeggs/rpmbuild/SRPMS/bodhi-3.13.0-0.0.beta.e0ca5bc.fc29.infra.src.rpm
|
|
....
|
|
|
|
When building a Bodhi release that is intended for production, we should
|
|
build from the production dist-git repo instead of uploading an SRPM:
|
|
|
|
....
|
|
$ koji build f29-infra git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37
|
|
....
|
|
|
|
All builds against the `f29-infra` build target will go into the
|
|
`f29-infra-stg` repository. If you wish to promote a build from staging
|
|
to production, you can do something like this command:
|
|
|
|
....
|
|
$ koji move-build f29-infra-stg f29-infra bodhi-3.13.0-1.fc29.infra
|
|
....
|
|
|
|
=== Staging
|
|
|
|
The upgrade playbook will apply configuration changes after running the
|
|
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
|
systems in order to get the upgrade playbook to succeed. If you are in
|
|
this situation, you can apply those changes by running the bodhi-backend
|
|
playbook:
|
|
|
|
....
|
|
sudo rbac-playbook -l staging groups/bodhi-backend.yml
|
|
....
|
|
|
|
In the
|
|
https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[os_masters inventory],
|
|
edit the `bodhi_version` setting to the version you wish to deploy to
|
|
staging. For example, to deploy `bodhi-3.13.0-1.fc29.infra` to staging,
|
|
I would set that varible like this:
|
|
|
|
....
|
|
bodhi_version: "bodhi-3.13.0-1.fc29.infra"
|
|
....
|
|
|
|
Run these commands:
|
|
|
|
....
|
|
# Synchronize the database from production to staging
|
|
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
|
|
# Upgrade the Bodhi backend on staging
|
|
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging
|
|
# Upgrade the Bodhi frontend on staging
|
|
$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging
|
|
....
|
|
|
|
=== Production
|
|
|
|
The upgrade playbook will apply configuration changes after running the
|
|
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
|
systems in order to get the upgrade playbook to succeed. If you are in
|
|
this situation, you can apply those changes by running the bodhi-backend
|
|
playbook:
|
|
|
|
....
|
|
sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend
|
|
....
|
|
|
|
In the
|
|
https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[os_masters inventory],
|
|
edit the `bodhi_version` setting to the version you wish to deploy to
|
|
production. For example, to deploy `bodhi-3.13.0-1.fc29.infra` to
|
|
production, I would set that varible like this:
|
|
|
|
....
|
|
bodhi_version: "bodhi-3.13.0-1.fc29.infra"
|
|
....
|
|
|
|
To update the bodhi RPMs in production:
|
|
|
|
....
|
|
# Update the backend VMs (this will also run the migrations, if any)
|
|
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend
|
|
# Update the frontend
|
|
$ sudo rbac-playbook openshift-apps/bodhi.yml
|
|
....
|
|
|
|
== Syncing the production database to staging
|
|
|
|
This can be useful for testing issues with production data in staging:
|
|
|
|
....
|
|
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
|
|
....
|
|
|
|
== Release EOL
|
|
|
|
....
|
|
bodhi-manage-releases edit --name F21 --state archived
|
|
....
|
|
|
|
== Adding notices to the front page or new update form
|
|
|
|
You can easily add notification messages to the front page of bodhi
|
|
using the _frontpage_notice_ option in
|
|
_ansible/roles/bodhi2/base/templates/production.ini.j2_. If
|
|
you want to flash a message on the New Update Form, you can use the
|
|
_newupdate_notice_ variable instead. This can be useful for
|
|
announcing things like service outages, etc.
|
|
|
|
== Using the Bodhi Shell to modify updates by hand
|
|
|
|
The "bodhi shell" is a Python shell with the SQLAlchemy session and
|
|
transaction manager initialized. It can be run from any
|
|
production/staging backend instance and allows you to modify any models
|
|
by hand.
|
|
|
|
....
|
|
sudo pshell /etc/bodhi/production.ini
|
|
|
|
# Execute a script that sets up the `db` and provides a `delete_update` function.
|
|
# This will eventually be shipped in the bodhi package, but can also be found here.
|
|
# https://raw.githubusercontent.com/fedora-infra/bodhi/develop/tools/shelldb.py
|
|
>>> execfile('shelldb.py')
|
|
....
|
|
|
|
At this point you have access to a _db_ SQLAlchemy Session
|
|
instance, a _t_ _transaction module_, and
|
|
_m_ for the _bodhi.models_.
|
|
|
|
....
|
|
# Fetch an update, and tweak it as necessary.
|
|
>>> up = m.Update.get(u'u'FEDORA-2016-4d226a5f7e', db)
|
|
|
|
# Commit the transaction
|
|
>>> t.commit()
|
|
....
|
|
|
|
Here is an example of merging two updates together and deleting the
|
|
original.
|
|
|
|
....
|
|
>>> up = m.Update.get(u'FEDORA-2016-4d226a5f7e', db)
|
|
>>> up.builds
|
|
[<Build {'epoch': 0, 'nvr': u'resteasy-3.0.17-2.fc24'}>, <Build {'epoch': 0, 'nvr': u'pki-core-10.3.5-1.fc24'}>]
|
|
>>> b = up.builds[0]
|
|
>>> up2 = m.Update.get(u'FEDORA-2016-5f63a874ca', db)
|
|
>>> up2.builds
|
|
[<Build {'epoch': 0, 'nvr': u'resteasy-3.0.17-3.fc24'}>]
|
|
>>> up.builds.remove(b)
|
|
>>> up.builds.append(up2.builds[0])
|
|
>>> delete_update(up2)
|
|
>>> t.commit()
|
|
....
|
|
|
|
== Using the Bodhi shell to fix uniqueness problems with e-mail addresses
|
|
|
|
Bodhi currently enforces uniqueness on user e-mail addresses. There is
|
|
https://github.com/fedora-infra/bodhi/issues/2387[an issue] filed to
|
|
drop this upstream, but for the time being the constraint is enforced.
|
|
This can be a problem for users who have more than one FAS account if
|
|
they make one account use an e-mail address that was previously used by
|
|
another account, if that other account has not logged into Bodhi since
|
|
it was changed to use a different address. One way the user can fix this
|
|
themselves is to log in to Bodhi with the old account so that Bodhi
|
|
learns about its new address. However, an admin can also fix this by
|
|
hand by using the Bodhi shell.
|
|
|
|
For example, suppose a user has created `user_1` and `user_2`. Suppose
|
|
that `user_1` used to use `email_a@example.com` but has been changed to
|
|
use `email_b@example.com` in FAS, and `user_2` is now configured to use
|
|
`email_a@example.com` in FAS. If `user_2` attempts to log in to Bodhi,
|
|
it will cause a uniqueness violation since Bodhi does not know that
|
|
`user_1` has changed to `email_b@example.com`. The user can simply log
|
|
in as `user_1` to fix this, which will cause Bodhi to update its e-mail
|
|
address to `email_b@example.com`. Or an admin can fix it with a shell on
|
|
one of the Bodhi backend servers like this:
|
|
|
|
....
|
|
[bowlofeggs@bodhi-backend02 ~][PROD]$ sudo -u apache pshell /etc/bodhi/production.ini
|
|
2018-05-29 20:21:36,366 INFO [bodhi][MainThread] Using python-bugzilla
|
|
2018-05-29 20:21:36,367 DEBUG [bodhi][MainThread] Using Koji Buildsystem
|
|
2018-05-29 20:21:42,559 INFO [bodhi.server][MainThread] Bodhi ready and at your service!
|
|
Python 2.7.14 (default, Mar 14 2018, 13:36:31)
|
|
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
|
|
Type "help" for more information.
|
|
|
|
Environment:
|
|
app The WSGI application.
|
|
registry Active Pyramid registry.
|
|
request Active request object.
|
|
root Root of the default resource tree.
|
|
root_factory Default root factory used to create `root`.
|
|
|
|
Custom Variables:
|
|
m bodhi.server.models
|
|
|
|
>>> u = m.User.query.filter_by(name=u'user_1').one()
|
|
>>> u.email = u'email_b@example.com'
|
|
>>> m.Session().commit()
|
|
....
|
|
|
|
== Troubleshooting and Resolution
|
|
|
|
=== Atomic OSTree compose failure
|
|
|
|
If the Atomic OSTree compose fails with some sort of
|
|
_Device or Resource busy_ error, then run _mount_ to see if there
|
|
are any stray _tmpfs_ mounts still active:
|
|
|
|
....
|
|
tmpfs on /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq type tmpfs (rw,relatime,seclabel,mode=755)
|
|
....
|
|
|
|
You can then
|
|
_umount /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq_
|
|
and resume the push again.
|
|
|
|
=== nfs repodata cache IOError
|
|
|
|
Sometimes you may hit an IOError during the updateinfo.xml generation
|
|
process from createrepo_c:
|
|
|
|
....
|
|
IOError: Cannot open /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml: File /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml doesn't exists or not a regular file
|
|
....
|
|
|
|
This issue will be resolved with NFSv4, but in the mean time it can be
|
|
worked around by removing the _.repocache_ directory and
|
|
resuming the push:
|
|
|
|
....
|
|
rm -fr /mnt/koji/mash/updates/epel7.repocache
|
|
....
|