Added the infra SOPs ported to asciidoc.
This commit is contained in:
Adam Saleh
parent
8a7f111a12
commit
a0301e30f1
148 changed files with 18575 additions and 17 deletions
429
modules/sysadmin_guide/pages/bodhi.adoc
Normal file
429
modules/sysadmin_guide/pages/bodhi.adoc
Normal file
|
|
@ -0,0 +1,429 @@
|
|||
= Bodhi Infrastructure SOP
|
||||
|
||||
Bodhi is used by Fedora developers to submit potential package updates
|
||||
for releases and to manage buildroot overrides. From here, bodhi handles
|
||||
all of the dirty work, from sending around emails, dealing with Koji, to
|
||||
composing the repositories.
|
||||
|
||||
Bodhi production instance: https://bodhi.fedoraproject.org Bodhi project
|
||||
page: https://github.com/fedora-infra/bodhi
|
||||
|
||||
== Contents
|
||||
|
||||
[arabic]
|
||||
. Contact Information
|
||||
. Adding a new pending release
|
||||
. 0-day Release Actions
|
||||
. Configuring all bodhi nodes
|
||||
. Pushing updates
|
||||
. Monitoring the bodhi composer output
|
||||
. Resuming a failed push
|
||||
. Performing a production bodhi upgrade
|
||||
. Syncing the production database to staging
|
||||
. Release EOL
|
||||
. Adding notices to the front page or new update form
|
||||
. Using the Bodhi Shell to modify updates by hand
|
||||
. Using the Bodhi shell to fix uniqueness problems with e-mail addresses
|
||||
. Troubleshooting and Resolution
|
||||
|
||||
== Contact Information
|
||||
|
||||
Owner::
|
||||
Fedora Infrastructure Team
|
||||
Contact::
|
||||
#fedora-admin
|
||||
Persons::
|
||||
bowlofeggs
|
||||
Location::
|
||||
Phoenix
|
||||
Servers::
|
||||
* bodhi-backend01.phx2.fedoraproject.org (composer)
|
||||
* os.fedoraproject.org (web front end and backend task workers for
|
||||
non-compose tasks)
|
||||
* bodhi-backend01.stg.phx2.fedoraproject.org (staging composer)
|
||||
* os.stg.fedoraproject.org (staging web front end and backend task
|
||||
workers for non-compose tasks)
|
||||
Purpose::
|
||||
Push package updates, and handle new submissions.
|
||||
|
||||
== Adding a new pending release
|
||||
|
||||
Adding and modifying releases is done using the
|
||||
[.title-ref]#bodhi-manage-releases# tool.
|
||||
|
||||
You can add a new pending release by running this command:
|
||||
|
||||
....
|
||||
bodhi-manage-releases create --name F23 --long-name "Fedora 23" --id-prefix FEDORA --version 23 --branch f23 --dist-tag f23 --stable-tag f23-updates --testing-tag f23-updates-testing --candidate-tag f23-updates-candidate --pending-stable-tag f23-updates-pending --pending-testing-tag f23-updates-testing-pending --override-tag f23-override --state pending
|
||||
....
|
||||
|
||||
== Pre-Beta Bodhi config
|
||||
|
||||
Enable pre_beta policy in bodhi config in ansible.::::
|
||||
ansible/roles/bodhi2/base/templates/production.ini.j2
|
||||
|
||||
Uncomment or add the following lines:
|
||||
|
||||
....
|
||||
#f29.status = pre_beta
|
||||
#f29.pre_beta.mandatory_days_in_testing = 3
|
||||
#f29.pre_beta.critpath.min_karma = 1
|
||||
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
|
||||
....
|
||||
|
||||
== Post-Beta Bodhi config
|
||||
|
||||
Enable post_beta policy in bodhi config in ansible.::::
|
||||
ansible/roles/bodhi2/base/templates/production.ini.j2
|
||||
|
||||
Comment or remove the following lines corresponding to pre_beta policy:
|
||||
|
||||
....
|
||||
#f29.status = pre_beta
|
||||
#f29.pre_beta.mandatory_days_in_testing = 3
|
||||
#f29.pre_beta.critpath.min_karma = 1
|
||||
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
|
||||
....
|
||||
|
||||
Uncomment or add the following lines for post_beta policy
|
||||
|
||||
....
|
||||
#f29.status = post_beta
|
||||
#f29.post_beta.mandatory_days_in_testing = 7
|
||||
#f29.post_beta.critpath.min_karma = 2
|
||||
#f29.post_beta.critpath.stable_after_days_without_negative_karma = 14
|
||||
....
|
||||
|
||||
== 0-day Release Actions
|
||||
|
||||
* update atomic config
|
||||
* run the ansible playbook
|
||||
|
||||
Going from pending to a proper release in bodhi requires a few steps:
|
||||
|
||||
Change state from pending to current:
|
||||
|
||||
....
|
||||
bodhi-manage-releases edit --name F23 --state current
|
||||
....
|
||||
|
||||
You may also need to disable any pre-beta or post-beta policy defined in
|
||||
the bodhi config in ansible.:
|
||||
|
||||
....
|
||||
ansible/roles/bodhi2/base/templates/production.ini.j2
|
||||
....
|
||||
|
||||
Uncomment or remove the lines related to pre and post beta polcy
|
||||
|
||||
....
|
||||
#f29.status = post_beta
|
||||
#f29.post_beta.mandatory_days_in_testing = 7
|
||||
#f29.post_beta.critpath.min_karma = 2
|
||||
#f29.post_beta.critpath.stable_after_days_without_negative_karma = 14
|
||||
#f29.status = pre_beta
|
||||
#f29.pre_beta.mandatory_days_in_testing = 3
|
||||
#f29.pre_beta.critpath.min_karma = 1
|
||||
#f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14
|
||||
....
|
||||
|
||||
== Configuring all bodhi nodes
|
||||
|
||||
Run this command from the [.title-ref]#ansible# checkout to configure
|
||||
all of bodhi in production:
|
||||
|
||||
....
|
||||
# This will configure the backends
|
||||
$ sudo rbac-playbook playbooks/groups/bodhi2.yml
|
||||
# This will configure the frontend
|
||||
$ sudo rbac-playbook openshift-apps/bodhi.yml
|
||||
....
|
||||
|
||||
== Pushing updates
|
||||
|
||||
SSH into the [.title-ref]#bodhi-backend01# machine and run:
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push
|
||||
....
|
||||
|
||||
You can restrict the updates by release and/or request:
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push --releases f23,f22 --request stable
|
||||
....
|
||||
|
||||
You can also push specific builds:
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push --builds openssl-1.0.1k-14.fc22,openssl-1.0.1k-14.fc23
|
||||
....
|
||||
|
||||
This will display a list of updates that are ready to be pushed.
|
||||
|
||||
== Monitoring the bodhi composer output
|
||||
|
||||
You can monitor the bodhi composer via the `bodhi` CLI tool, or via the
|
||||
systemd journal on `bodhi-backend01`:
|
||||
|
||||
....
|
||||
# From the comfort of your own laptop.
|
||||
$ bodhi composes list
|
||||
# From bodhi-backend01
|
||||
$ journalctl -f -u fedmsg-hub
|
||||
....
|
||||
|
||||
== Resuming a failed push
|
||||
|
||||
If a push fails for some reason, you can easily resume it on
|
||||
`bodhi-backend01` by running:
|
||||
|
||||
....
|
||||
$ sudo -u apache bodhi-push --resume
|
||||
....
|
||||
|
||||
== Performing a bodhi upgrade
|
||||
|
||||
=== Build Bodhi
|
||||
|
||||
Bodhi is deployed from the infrastructure Koji repositories. At the time
|
||||
of this writing, it is deployed from the `f29-infra` and `f29-infra-stg`
|
||||
(for staging) repositories. Bodhi is built for these repositories from
|
||||
the `master` branch of the
|
||||
https://src.fedoraproject.org/rpms/bodhi[bodhi dist-git repository].
|
||||
|
||||
As an example, to build a Bodhi beta for the `f29-infra-stg` repository,
|
||||
you can use this command:
|
||||
|
||||
....
|
||||
$ rpmbuild --define "dist .fc29.infra" -bs bodhi.spec
|
||||
Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-3.13.0-0.0.beta.e0ca5bc.fc29.infra.src.rpm
|
||||
$ koji build f29-infra /home/bowlofeggs/rpmbuild/SRPMS/bodhi-3.13.0-0.0.beta.e0ca5bc.fc29.infra.src.rpm
|
||||
....
|
||||
|
||||
When building a Bodhi release that is intended for production, we should
|
||||
build from the production dist-git repo instead of uploading an SRPM:
|
||||
|
||||
....
|
||||
$ koji build f29-infra git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37
|
||||
....
|
||||
|
||||
All builds against the `f29-infra` build target will go into the
|
||||
`f29-infra-stg` repository. If you wish to promote a build from staging
|
||||
to production, you can do something like this command:
|
||||
|
||||
....
|
||||
$ koji move-build f29-infra-stg f29-infra bodhi-3.13.0-1.fc29.infra
|
||||
....
|
||||
|
||||
=== Staging
|
||||
|
||||
The upgrade playbook will apply configuration changes after running the
|
||||
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
||||
systems in order to get the upgrade playbook to succeed. If you are in
|
||||
this situation, you can apply those changes by running the bodhi-backend
|
||||
playbook:
|
||||
|
||||
....
|
||||
sudo rbac-playbook -l staging groups/bodhi-backend.yml
|
||||
....
|
||||
|
||||
In the [.title-ref]#os_masters inventory#
|
||||
<https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg>_,
|
||||
edit the `bodhi_version` setting it to the version you wish to deploy to
|
||||
staging. For example, to deploy `bodhi-3.13.0-1.fc29.infra` to staging,
|
||||
I would set that varible like this:
|
||||
|
||||
....
|
||||
bodhi_version: "bodhi-3.13.0-1.fc29.infra"
|
||||
....
|
||||
|
||||
Run these commands:
|
||||
|
||||
....
|
||||
# Synchronize the database from production to staging
|
||||
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
|
||||
# Upgrade the Bodhi backend on staging
|
||||
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging
|
||||
# Upgrade the Bodhi frontend on staging
|
||||
$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging
|
||||
....
|
||||
|
||||
=== Production
|
||||
|
||||
The upgrade playbook will apply configuration changes after running the
|
||||
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
||||
systems in order to get the upgrade playbook to succeed. If you are in
|
||||
this situation, you can apply those changes by running the bodhi-backend
|
||||
playbook:
|
||||
|
||||
....
|
||||
sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend
|
||||
....
|
||||
|
||||
In the [.title-ref]#os_masters inventory#
|
||||
<https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters>_,
|
||||
edit the `bodhi_version` setting it to the version you wish to deploy to
|
||||
production. For example, to deploy `bodhi-3.13.0-1.fc29.infra` to
|
||||
production, I would set that varible like this:
|
||||
|
||||
....
|
||||
bodhi_version: "bodhi-3.13.0-1.fc29.infra"
|
||||
....
|
||||
|
||||
To update the bodhi RPMs in production:
|
||||
|
||||
....
|
||||
# Update the backend VMs (this will also run the migrations, if any)
|
||||
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend
|
||||
# Update the frontend
|
||||
$ sudo rbac-playbook openshift-apps/bodhi.yml
|
||||
....
|
||||
|
||||
== Syncing the production database to staging
|
||||
|
||||
This can be useful for testing issues with production data in staging:
|
||||
|
||||
....
|
||||
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
|
||||
....
|
||||
|
||||
== Release EOL
|
||||
|
||||
....
|
||||
bodhi-manage-releases edit --name F21 --state archived
|
||||
....
|
||||
|
||||
== Adding notices to the front page or new update form
|
||||
|
||||
You can easily add notification messages to the front page of bodhi
|
||||
using the [.title-ref]#frontpage_notice# option in
|
||||
[.title-ref]#ansible/roles/bodhi2/base/templates/production.ini.j2#. If
|
||||
you want to flash a message on the New Update Form, you can use the
|
||||
[.title-ref]#newupdate_notice# variable instead. This can be useful for
|
||||
announcing things like service outages, etc.
|
||||
|
||||
== Using the Bodhi Shell to modify updates by hand
|
||||
|
||||
The "bodhi shell" is a Python shell with the SQLAlchemy session and
|
||||
transaction manager initialized. It can be run from any
|
||||
production/staging backend instance and allows you to modify any models
|
||||
by hand.
|
||||
|
||||
....
|
||||
sudo pshell /etc/bodhi/production.ini
|
||||
|
||||
# Execute a script that sets up the `db` and provides a `delete_update` function.
|
||||
# This will eventually be shipped in the bodhi package, but can also be found here.
|
||||
# https://raw.githubusercontent.com/fedora-infra/bodhi/develop/tools/shelldb.py
|
||||
>>> execfile('shelldb.py')
|
||||
....
|
||||
|
||||
At this point you have access to a [.title-ref]#db# SQLAlchemy Session
|
||||
instance, a [.title-ref]#t# [.title-ref]#transaction# module, and
|
||||
[.title-ref]#m# for the [.title-ref]#bodhi.models#.
|
||||
|
||||
....
|
||||
# Fetch an update, and tweak it as necessary.
|
||||
>>> up = m.Update.get(u'u'FEDORA-2016-4d226a5f7e', db)
|
||||
|
||||
# Commit the transaction
|
||||
>>> t.commit()
|
||||
....
|
||||
|
||||
Here is an example of merging two updates together and deleting the
|
||||
original.
|
||||
|
||||
....
|
||||
>>> up = m.Update.get(u'FEDORA-2016-4d226a5f7e', db)
|
||||
>>> up.builds
|
||||
[<Build {'epoch': 0, 'nvr': u'resteasy-3.0.17-2.fc24'}>, <Build {'epoch': 0, 'nvr': u'pki-core-10.3.5-1.fc24'}>]
|
||||
>>> b = up.builds[0]
|
||||
>>> up2 = m.Update.get(u'FEDORA-2016-5f63a874ca', db)
|
||||
>>> up2.builds
|
||||
[<Build {'epoch': 0, 'nvr': u'resteasy-3.0.17-3.fc24'}>]
|
||||
>>> up.builds.remove(b)
|
||||
>>> up.builds.append(up2.builds[0])
|
||||
>>> delete_update(up2)
|
||||
>>> t.commit()
|
||||
....
|
||||
|
||||
== Using the Bodhi shell to fix uniqueness problems with e-mail addresses
|
||||
|
||||
Bodhi currently enforces uniqueness on user e-mail addresses. There is
|
||||
https://github.com/fedora-infra/bodhi/issues/2387[an issue] filed to
|
||||
drop this upstream, but for the time being the constraint is enforced.
|
||||
This can be a problem for users who have more than one FAS account if
|
||||
they make one account use an e-mail address that was previously used by
|
||||
another account, if that other account has not logged into Bodhi since
|
||||
it was changed to use a different address. One way the user can fix this
|
||||
themselves is to log in to Bodhi with the old account so that Bodhi
|
||||
learns about its new address. However, an admin can also fix this by
|
||||
hand by using the Bodhi shell.
|
||||
|
||||
For example, suppose a user has created `user_1` and `user_2`. Suppose
|
||||
that `user_1` used to use `email_a@example.com` but has been changed to
|
||||
use `email_b@example.com` in FAS, and `user_2` is now configured to use
|
||||
`email_a@example.com` in FAS. If `user_2` attempts to log in to Bodhi,
|
||||
it will cause a uniqueness violation since Bodhi does not know that
|
||||
`user_1` has changed to `email_b@example.com`. The user can simply log
|
||||
in as `user_1` to fix this, which will cause Bodhi to update its e-mail
|
||||
address to `email_b@example.com`. Or an admin can fix it with a shell on
|
||||
one of the Bodhi backend servers like this:
|
||||
|
||||
....
|
||||
[bowlofeggs@bodhi-backend02 ~][PROD]$ sudo -u apache pshell /etc/bodhi/production.ini
|
||||
2018-05-29 20:21:36,366 INFO [bodhi][MainThread] Using python-bugzilla
|
||||
2018-05-29 20:21:36,367 DEBUG [bodhi][MainThread] Using Koji Buildsystem
|
||||
2018-05-29 20:21:42,559 INFO [bodhi.server][MainThread] Bodhi ready and at your service!
|
||||
Python 2.7.14 (default, Mar 14 2018, 13:36:31)
|
||||
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
|
||||
Type "help" for more information.
|
||||
|
||||
Environment:
|
||||
app The WSGI application.
|
||||
registry Active Pyramid registry.
|
||||
request Active request object.
|
||||
root Root of the default resource tree.
|
||||
root_factory Default root factory used to create `root`.
|
||||
|
||||
Custom Variables:
|
||||
m bodhi.server.models
|
||||
|
||||
>>> u = m.User.query.filter_by(name=u'user_1').one()
|
||||
>>> u.email = u'email_b@example.com'
|
||||
>>> m.Session().commit()
|
||||
....
|
||||
|
||||
== Troubleshooting and Resolution
|
||||
|
||||
=== Atomic OSTree compose failure
|
||||
|
||||
If the Atomic OSTree compose fails with some sort of [.title-ref]#Device
|
||||
or Resource busy# error, then run [.title-ref]#mount# to see if there
|
||||
are any stray [.title-ref]#tmpfs# mounts still active:
|
||||
|
||||
....
|
||||
tmpfs on /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq type tmpfs (rw,relatime,seclabel,mode=755)
|
||||
....
|
||||
|
||||
You can then [.title-ref]#umount
|
||||
/var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq#
|
||||
and resume the push again.
|
||||
|
||||
=== nfs repodata cache IOError
|
||||
|
||||
Sometimes you may hit an IOError during the updateinfo.xml generation
|
||||
process from createrepo_c:
|
||||
|
||||
....
|
||||
IOError: Cannot open /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml: File /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml doesn't exists or not a regular file
|
||||
....
|
||||
|
||||
This issue will be resolved with NFSv4, but in the mean time it can be
|
||||
worked around by removing the [.title-ref]#.repocache# directory and
|
||||
resuming the push:
|
||||
|
||||
....
|
||||
rm -fr /mnt/koji/mash/updates/epel7.repocache
|
||||
....
|
||||
Loading…
Add table
Add a link
Reference in a new issue