543 lines
18 KiB
Text
543 lines
18 KiB
Text
= Bodhi Infrastructure SOP
|
||
|
||
Bodhi is used by Fedora developers to submit potential package updates
|
||
for releases and to manage buildroot overrides. From here, bodhi handles
|
||
all of the dirty work, from sending around emails, dealing with Koji, to
|
||
composing the repositories.
|
||
|
||
Bodhi production instance: https://bodhi.fedoraproject.org
|
||
|
||
Bodhi project page: https://github.com/fedora-infra/bodhi
|
||
|
||
== Contents
|
||
|
||
* <<_contact_information>>
|
||
* <<_adding_a_new_pending_release>>
|
||
* <<_pre_beta_bodhi_config>>
|
||
* <<_post_beta_bodhi_config>>
|
||
* <<_0_day_release_actions>>
|
||
* <<_configuring_all_bodhi_nodes>>
|
||
* <<_pushing_updates>>
|
||
* <<_monitoring_the_bodhi_composer_output>>
|
||
* <<_resuming_a_failed_push>>
|
||
* <<_performing_a_bodhi_upgrade>>
|
||
* <<_syncing_the_production_database_to_staging>>
|
||
* <<_release_eol>>
|
||
* <<_adding_notices_to_the_front_page_or_new_update_form>>
|
||
* <<_using_the_bodhi_shell_to_modify_updates_by_hand>>
|
||
* <<_using_the_bodhi_shell_to_fix_uniqueness_problems_with_e_mail_addresses>>
|
||
* <<_troubleshooting_and_resolution>>
|
||
|
||
== Contact Information
|
||
|
||
Owner::
|
||
Fedora Infrastructure Team
|
||
Contact::
|
||
#fedora-admin
|
||
Persons::
|
||
bowlofeggs
|
||
Location::
|
||
iad2
|
||
Servers::
|
||
* bodhi-backend01.iad2.fedoraproject.org (composer)
|
||
* os.fedoraproject.org (web front end and backend task workers for
|
||
non-compose tasks)
|
||
* bodhi-backend01.stg.iad2.fedoraproject.org (staging composer)
|
||
* os.stg.fedoraproject.org (staging web front end and backend task
|
||
workers for non-compose tasks)
|
||
Purpose::
|
||
Push package updates, and handle new submissions.
|
||
|
||
== Adding a new pending release
|
||
|
||
Adding and modifying releases is done using the
|
||
_bodhi-manage-releases_ tool.
|
||
|
||
You can add a new pending release by running this command:
|
||
|
||
....
|
||
bodhi-manage-releases create --name F23 --long-name "Fedora 23" --id-prefix FEDORA --version 23 --branch rawhide --dist-tag f23 --stable-tag f23-updates --testing-tag f23-updates-testing --candidate-tag f23-updates-candidate --pending-stable-tag f23-updates-pending --pending-testing-tag f23-updates-testing-pending --override-tag f23-override --package-manager dnf --testing-repository updates-testing --state pending --not-composed-by-bodhi --create-automatic-updates
|
||
....
|
||
|
||
== Pre-Beta Bodhi config
|
||
|
||
Disable automatic updates creation and enable composes by Bodhi:
|
||
|
||
....
|
||
bodhi-manage-releases edit --name F23 --composed-by-bodhi --no-create-automatic-updates --branch f23
|
||
....
|
||
|
||
Enable pre_beta policy in bodhi config in ansible.::::
|
||
ansible/roles/bodhi2/base/templates/production.ini.j2
|
||
|
||
Uncomment or add the following lines:
|
||
|
||
....
|
||
#f29.status = pre_beta
|
||
#f29.pre_beta.mandatory_days_in_testing = 3
|
||
#f29.pre_beta.critpath.min_karma = 1
|
||
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
|
||
....
|
||
|
||
== Post-Beta Bodhi config
|
||
|
||
Enable post_beta policy in bodhi config in ansible.::::
|
||
ansible/roles/bodhi2/base/templates/production.ini.j2
|
||
|
||
Comment or remove the following lines corresponding to pre_beta policy:
|
||
|
||
....
|
||
#f29.status = pre_beta
|
||
#f29.pre_beta.mandatory_days_in_testing = 3
|
||
#f29.pre_beta.critpath.min_karma = 1
|
||
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
|
||
....
|
||
|
||
Uncomment or add the following lines for post_beta policy
|
||
|
||
....
|
||
#f29.status = post_beta
|
||
#f29.post_beta.mandatory_days_in_testing = 7
|
||
#f29.post_beta.critpath.min_karma = 2
|
||
#f29.post_beta.critpath.stable_after_days_without_negative_karma = 14
|
||
....
|
||
|
||
== 0-day Release Actions
|
||
|
||
* update atomic config
|
||
* run the ansible playbook
|
||
|
||
Going from pending to a proper release in bodhi requires a few steps:
|
||
|
||
Change state from pending to current:
|
||
|
||
....
|
||
bodhi-manage-releases edit --name F23 --state current
|
||
....
|
||
|
||
You may also need to disable any pre-beta or post-beta policy defined in
|
||
the bodhi config in ansible.:
|
||
|
||
....
|
||
ansible/roles/bodhi2/base/templates/production.ini.j2
|
||
....
|
||
|
||
Uncomment or remove the lines related to pre and post beta polcy
|
||
|
||
....
|
||
#f29.status = post_beta
|
||
#f29.post_beta.mandatory_days_in_testing = 7
|
||
#f29.post_beta.critpath.min_karma = 2
|
||
#f29.post_beta.critpath.stable_after_days_without_negative_karma = 14
|
||
#f29.status = pre_beta
|
||
#f29.pre_beta.mandatory_days_in_testing = 3
|
||
#f29.pre_beta.critpath.min_karma = 1
|
||
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
|
||
....
|
||
|
||
== Configuring all bodhi nodes
|
||
|
||
Run this command from the _ansible_ checkout to configure
|
||
all of bodhi in production:
|
||
|
||
....
|
||
# This will configure the backends
|
||
$ sudo rbac-playbook playbooks/groups/bodhi2.yml
|
||
# This will configure the frontend
|
||
$ sudo rbac-playbook openshift-apps/bodhi.yml
|
||
....
|
||
|
||
== Pushing updates
|
||
|
||
SSH into the `bodhi-backend01` machine and run:
|
||
|
||
....
|
||
$ sudo -u apache bodhi-push
|
||
....
|
||
|
||
You can restrict the updates by release and/or request:
|
||
|
||
....
|
||
$ sudo -u apache bodhi-push --releases f23,f22 --request stable
|
||
....
|
||
|
||
You can also push specific builds:
|
||
|
||
....
|
||
$ sudo -u apache bodhi-push --builds openssl-1.0.1k-14.fc22,openssl-1.0.1k-14.fc23
|
||
....
|
||
|
||
This will display a list of updates that are ready to be pushed.
|
||
|
||
== Monitoring the bodhi composer output
|
||
|
||
You can monitor the bodhi composer via the `bodhi` CLI tool, or via the
|
||
systemd journal on `bodhi-backend01`:
|
||
|
||
....
|
||
# From the comfort of your own laptop.
|
||
$ bodhi composes list
|
||
# From bodhi-backend01
|
||
$ journalctl -f -u fedmsg-hub
|
||
....
|
||
|
||
== Resuming a failed push
|
||
|
||
If a push fails for some reason, you can easily resume it on
|
||
`bodhi-backend01` by running:
|
||
|
||
....
|
||
$ sudo -u apache bodhi-push --resume
|
||
....
|
||
|
||
== Performing a bodhi upgrade
|
||
|
||
==== Have the release in the official Bodhi repository ready
|
||
|
||
Follow steps in
|
||
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst_]
|
||
up to the point of building a beta-release.
|
||
|
||
Changing the specfile manually is tedious and error-prone, it is more
|
||
useful to just do several patch releases on staging, before we commit to
|
||
one to deploy on production.
|
||
|
||
==== Testing deployed Bodhi without making a release
|
||
|
||
If the Bodhi changes don’t involve any of
|
||
|
||
* Composer code
|
||
* Doesn’t include any upgrades to database with alembic
|
||
|
||
You can opt to use the hotfix-style container-image that can be found
|
||
among the templates:
|
||
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix_]
|
||
|
||
Change the env-vars in the docker-file-base-hotfix to reflect the commit
|
||
to be built. The image definition is not buildable locally, as it needs
|
||
the openshift environment to configure koji and kerberos.
|
||
|
||
Change the buildconfig to point to the new definition:
|
||
|
||
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14_]
|
||
|
||
I.e:
|
||
|
||
[cols=",",]
|
||
|===
|
||
| |\{% if env == "staging" %}
|
||
| | \{\{ load_file('dockerfile-base-hotfix') | indent(8) }}
|
||
| |\{% else %}
|
||
| | \{\{ load_file('dockerfile-base') | indent(8) }}
|
||
| |\{% endif %}
|
||
|===
|
||
|
||
Then commit and push the changes, to be able to deploy from batcave
|
||
with:
|
||
|
||
sudo rbac-playbook openshift-apps/bodhi.yml -l staging
|
||
|
||
==== Build Bodhi
|
||
|
||
Bodhi is deployed from the infrastructure Koji repositories. At the time
|
||
of this writing, it is deployed from the f34-infra and f34-infra-stg
|
||
(for staging) repositories. Bodhi is built for these repositories from
|
||
the master branch of thehttps://src.fedoraproject.org/rpms/bodhi[
|
||
]https://src.fedoraproject.org/rpms/bodhi[_bodhi dist-git repository_].
|
||
|
||
As an example, to build a Bodhi for the f34-infra-stg repository, you
|
||
can use this command:
|
||
|
||
....
|
||
$ rpmbuild --define "dist .f34.infra" -bs bodhi.spec
|
||
....
|
||
|
||
Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
|
||
|
||
....
|
||
$ koji build f34-infra
|
||
/home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
|
||
....
|
||
|
||
When building a Bodhi release that is intended for production, we should
|
||
build from the production dist-git repo instead of uploading an SRPM:
|
||
|
||
$ koji build f34-infra
|
||
git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37
|
||
|
||
All builds against the f34-infra build target will go into the
|
||
f34-infra-stg repository. If you wish to promote a build from staging to
|
||
production, you can do something like this command:
|
||
|
||
$ koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.f34.infra
|
||
|
||
=== Staging
|
||
|
||
The upgrade playbook will apply configuration changes after running the
|
||
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
||
systems in order to get the upgrade playbook to succeed. If you are in
|
||
this situation, you can apply those changes by running the bodhi-backend
|
||
playbook:
|
||
|
||
....
|
||
sudo rbac-playbook -l staging groups/bodhi-backend.yml
|
||
....
|
||
|
||
In
|
||
thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[
|
||
]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[_os_masters
|
||
inventory_], edit the bodhi_version setting to the version you wish to
|
||
deploy to staging. For example, to deploy version 5.7.3 to
|
||
staging, I would set that varible like this:
|
||
|
||
bodhi_version: ‘5.7.3’
|
||
|
||
The package itself will then be installed in container build by ‘dnf
|
||
install -y bodhi-server-\{\{bodhi_version}}’
|
||
|
||
Beware, when running the playbooks on the backend vm, it currently
|
||
installs the latest bodhi package available in the respective
|
||
infra/infra-stg tag.
|
||
|
||
It can be useful to verify correct version is available on the backend,
|
||
i.e. for staging run
|
||
|
||
....
|
||
ssh bodhi-backend01.stg.iad2.fedoraproject.org
|
||
....
|
||
|
||
....
|
||
dnf list bodhi-server --refresh --all --showduplicates
|
||
....
|
||
|
||
*MAYBE WE SHOULD PIN THIS AS WELL!!!*
|
||
|
||
Run these commands:
|
||
|
||
....
|
||
# Synchronize the database from production to staging
|
||
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
|
||
# Upgrade the Bodhi backend on staging
|
||
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging
|
||
# Upgrade the Bodhi frontend on staging
|
||
$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging
|
||
....
|
||
|
||
==== Tests to run
|
||
|
||
For compose test you can follow
|
||
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta_]
|
||
|
||
For automated health-check, you can run
|
||
https://pagure.io/fedora-ci/monitor-gating[_https://pagure.io/fedora-ci/monitor-gating_],
|
||
|
||
It should be sufficient to check the openshift project
|
||
https://os.fedoraproject.org/console/project/monitor-gating[_https://os.fedoraproject.org/console/project/monitor-gating_]
|
||
where it is being run every few hours. There is a dashboard available
|
||
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/6q2qpcvGz/monitor-gating
|
||
|
||
==== Production
|
||
|
||
After testing in staging, you should be able to move the rpm from
|
||
staging to production:
|
||
|
||
....
|
||
koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.fc34
|
||
....
|
||
|
||
Even if the rpm version is pinned in your deploy, double check the
|
||
availability in the tag
|
||
|
||
The upgrade playbook will apply configuration changes after running the
|
||
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
||
systems in order to get the upgrade playbook to succeed. If you are in
|
||
this situation, you can apply those changes by running the bodhi-backend
|
||
playbook:
|
||
|
||
....
|
||
sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend
|
||
....
|
||
|
||
In
|
||
thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[
|
||
]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[_os_masters
|
||
inventory_], edit the bodhi_version setting to the version you wish to
|
||
deploy to production. For example, to deploy bodhi-5.7.3-0.f34.infra to
|
||
production, I would set that varible like this:
|
||
|
||
....
|
||
bodhi_version: "5.7.3"
|
||
....
|
||
|
||
To update the bodhi RPMs in production:
|
||
|
||
....
|
||
# Update the backend VMs (this will also run the migrations, if any)
|
||
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend
|
||
# Update the frontend
|
||
$ sudo rbac-playbook openshift-apps/bodhi.yml
|
||
....
|
||
|
||
All of the playbooks should finish with no failures.
|
||
|
||
Afterwards, check
|
||
https://os.fedoraproject.org/console/project/bodhi/overview[_https://os.fedoraproject.org/console/project/bodhi/overview_]
|
||
to see if the image has been built and if it has been properly deployed.
|
||
|
||
Things to watchout for
|
||
|
||
* Correct rpm not being available for container build
|
||
** This is the reason to check the infra-tag for bodhi
|
||
* Failure to start the pod due to backend db mismatch
|
||
** Sometimes the alembic upgrade that is supposed to run as part of
|
||
playbooks/manual/upgrade/bodhi.yml fails silently
|
||
* Some issues resurface only after few days in production, resulting in
|
||
visible stack-traces in logs.
|
||
|
||
=== Syncing the production database to staging
|
||
|
||
This can be useful for testing issues with production data in staging:
|
||
|
||
....
|
||
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
|
||
....
|
||
|
||
== Release EOL
|
||
|
||
....
|
||
bodhi-manage-releases edit --name F21 --state archived
|
||
....
|
||
|
||
== Adding notices to the front page or new update form
|
||
|
||
You can easily add notification messages to the front page of bodhi
|
||
using the _frontpage_notice_ option in
|
||
_ansible/roles/bodhi2/base/templates/production.ini.j2_. If
|
||
you want to flash a message on the New Update Form, you can use the
|
||
_newupdate_notice_ variable instead. This can be useful for
|
||
announcing things like service outages, etc.
|
||
|
||
== Using the Bodhi Shell to modify updates by hand
|
||
|
||
The "bodhi shell" is a Python shell with the SQLAlchemy session and
|
||
transaction manager initialized. It can be run from any
|
||
production/staging backend instance and allows you to modify any models
|
||
by hand.
|
||
|
||
....
|
||
sudo pshell /etc/bodhi/production.ini
|
||
|
||
# Execute a script that sets up the `db` and provides a `delete_update` function.
|
||
# This will eventually be shipped in the bodhi package, but can also be found here.
|
||
# https://raw.githubusercontent.com/fedora-infra/bodhi/develop/tools/shelldb.py
|
||
>>> execfile('shelldb.py')
|
||
....
|
||
|
||
At this point you have access to a _db_ SQLAlchemy Session
|
||
instance, a _t_ _transaction module_, and
|
||
_m_ for the _bodhi.models_.
|
||
|
||
....
|
||
# Fetch an update, and tweak it as necessary.
|
||
>>> up = m.Update.get(u'u'FEDORA-2016-4d226a5f7e', db)
|
||
|
||
# Commit the transaction
|
||
>>> t.commit()
|
||
....
|
||
|
||
Here is an example of merging two updates together and deleting the
|
||
original.
|
||
|
||
....
|
||
>>> up = m.Update.get(u'FEDORA-2016-4d226a5f7e', db)
|
||
>>> up.builds
|
||
[<Build {'epoch': 0, 'nvr': u'resteasy-3.0.17-2.fc24'}>, <Build {'epoch': 0, 'nvr': u'pki-core-10.3.5-1.fc24'}>]
|
||
>>> b = up.builds[0]
|
||
>>> up2 = m.Update.get(u'FEDORA-2016-5f63a874ca', db)
|
||
>>> up2.builds
|
||
[<Build {'epoch': 0, 'nvr': u'resteasy-3.0.17-3.fc24'}>]
|
||
>>> up.builds.remove(b)
|
||
>>> up.builds.append(up2.builds[0])
|
||
>>> delete_update(up2)
|
||
>>> t.commit()
|
||
....
|
||
|
||
== Using the Bodhi shell to fix uniqueness problems with e-mail addresses
|
||
|
||
Bodhi currently enforces uniqueness on user e-mail addresses. There is
|
||
https://github.com/fedora-infra/bodhi/issues/2387[an issue] filed to
|
||
drop this upstream, but for the time being the constraint is enforced.
|
||
This can be a problem for users who have more than one FAS account if
|
||
they make one account use an e-mail address that was previously used by
|
||
another account, if that other account has not logged into Bodhi since
|
||
it was changed to use a different address. One way the user can fix this
|
||
themselves is to log in to Bodhi with the old account so that Bodhi
|
||
learns about its new address. However, an admin can also fix this by
|
||
hand by using the Bodhi shell.
|
||
|
||
For example, suppose a user has created `user_1` and `user_2`. Suppose
|
||
that `user_1` used to use `email_a@example.com` but has been changed to
|
||
use `email_b@example.com` in FAS, and `user_2` is now configured to use
|
||
`email_a@example.com` in FAS. If `user_2` attempts to log in to Bodhi,
|
||
it will cause a uniqueness violation since Bodhi does not know that
|
||
`user_1` has changed to `email_b@example.com`. The user can simply log
|
||
in as `user_1` to fix this, which will cause Bodhi to update its e-mail
|
||
address to `email_b@example.com`. Or an admin can fix it with a shell on
|
||
one of the Bodhi backend servers like this:
|
||
|
||
....
|
||
[bowlofeggs@bodhi-backend02 ~][PROD]$ sudo -u apache pshell /etc/bodhi/production.ini
|
||
2018-05-29 20:21:36,366 INFO [bodhi][MainThread] Using python-bugzilla
|
||
2018-05-29 20:21:36,367 DEBUG [bodhi][MainThread] Using Koji Buildsystem
|
||
2018-05-29 20:21:42,559 INFO [bodhi.server][MainThread] Bodhi ready and at your service!
|
||
Python 2.7.14 (default, Mar 14 2018, 13:36:31)
|
||
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
|
||
Type "help" for more information.
|
||
|
||
Environment:
|
||
app The WSGI application.
|
||
registry Active Pyramid registry.
|
||
request Active request object.
|
||
root Root of the default resource tree.
|
||
root_factory Default root factory used to create `root`.
|
||
|
||
Custom Variables:
|
||
m bodhi.server.models
|
||
|
||
>>> u = m.User.query.filter_by(name=u'user_1').one()
|
||
>>> u.email = u'email_b@example.com'
|
||
>>> m.Session().commit()
|
||
....
|
||
|
||
== Troubleshooting and Resolution
|
||
|
||
=== Atomic OSTree compose failure
|
||
|
||
If the Atomic OSTree compose fails with some sort of
|
||
_Device or Resource busy_ error, then run _mount_ to see if there
|
||
are any stray _tmpfs_ mounts still active:
|
||
|
||
....
|
||
tmpfs on /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq type tmpfs (rw,relatime,seclabel,mode=755)
|
||
....
|
||
|
||
You can then
|
||
_umount /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq_
|
||
and resume the push again.
|
||
|
||
=== nfs repodata cache IOError
|
||
|
||
Sometimes you may hit an IOError during the updateinfo.xml generation
|
||
process from createrepo_c:
|
||
|
||
....
|
||
IOError: Cannot open /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml: File /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml doesn't exists or not a regular file
|
||
....
|
||
|
||
This issue will be resolved with NFSv4, but in the mean time it can be
|
||
worked around by removing the _.repocache_ directory and
|
||
resuming the push:
|
||
|
||
....
|
||
rm -fr /mnt/koji/mash/updates/epel7.repocache
|
||
....
|