= Bodhi Infrastructure Releng SOP Bodhi is used by Fedora developers to submit potential package updates for releases and to manage buildroot overrides. From here, bodhi handles all of the dirty work, from sending around emails, dealing with Koji, to composing the repositories. Bodhi production instance: https://bodhi.fedoraproject.org Bodhi project page: https://github.com/fedora-infra/bodhi == Contents * <<_contact_information>> * <<_adding_a_new_pending_release>> * <<_pre_beta_bodhi_config>> * <<_post_beta_bodhi_config>> * <<_0_day_release_actions>> * <<_configuring_all_bodhi_nodes>> * <<_pushing_updates>> * <<_monitoring_the_bodhi_composer_output>> * <<_resuming_a_failed_push>> * <<_performing_a_bodhi_upgrade>> * <<_syncing_the_production_database_to_staging>> * <<_release_eol>> * <<_adding_notices_to_the_front_page_or_new_update_form>> * <<_using_the_bodhi_shell_to_modify_updates_by_hand>> * <<_using_the_bodhi_shell_to_fix_uniqueness_problems_with_e_mail_addresses>> * <<_troubleshooting_and_resolution>> == Contact Information Owner:: Fedora Infrastructure Team Contact:: #fedora-admin Persons:: bowlofeggs Location:: iad2 Servers:: * bodhi-backend01.iad2.fedoraproject.org (composer) * os.fedoraproject.org (web front end and backend task workers for non-compose tasks) * bodhi-backend01.stg.iad2.fedoraproject.org (staging composer) * os.stg.fedoraproject.org (staging web front end and backend task workers for non-compose tasks) Purpose:: Push package updates, and handle new submissions. == Adding a new pending release Adding and modifying releases is done using the _bodhi-manage-releases_ tool. You can add a new pending release by running this command: .... bodhi-manage-releases create --name F23 --long-name "Fedora 23" --id-prefix FEDORA --version 23 --branch rawhide --dist-tag f23 --stable-tag f23-updates --testing-tag f23-updates-testing --candidate-tag f23-updates-candidate --pending-stable-tag f23-updates-pending --pending-testing-tag f23-updates-testing-pending --override-tag f23-override --package-manager dnf --testing-repository updates-testing --state pending --not-composed-by-bodhi --create-automatic-updates .... == Pre-Beta Bodhi config Disable automatic updates creation and enable composes by Bodhi: .... bodhi-manage-releases edit --name F23 --composed-by-bodhi --no-create-automatic-updates --branch f23 .... Enable pre_beta policy in bodhi config in ansible.:::: ansible/roles/bodhi2/base/templates/production.ini.j2 Uncomment or add the following lines: .... #f29.status = pre_beta #f29.pre_beta.mandatory_days_in_testing = 3 #f29.pre_beta.critpath.min_karma = 1 #f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14 .... == Post-Beta Bodhi config Enable post_beta policy in bodhi config in ansible.:::: ansible/roles/bodhi2/base/templates/production.ini.j2 Comment or remove the following lines corresponding to pre_beta policy: .... #f29.status = pre_beta #f29.pre_beta.mandatory_days_in_testing = 3 #f29.pre_beta.critpath.min_karma = 1 #f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14 .... Uncomment or add the following lines for post_beta policy .... #f29.status = post_beta #f29.post_beta.mandatory_days_in_testing = 7 #f29.post_beta.critpath.min_karma = 2 #f29.post_beta.critpath.stable_after_days_without_negative_karma = 14 .... == 0-day Release Actions * update atomic config * run the ansible playbook Going from pending to a proper release in bodhi requires a few steps: Change state from pending to current: .... bodhi-manage-releases edit --name F23 --state current .... You may also need to disable any pre-beta or post-beta policy defined in the bodhi config in ansible.: .... ansible/roles/bodhi2/base/templates/production.ini.j2 .... Uncomment or remove the lines related to pre and post beta polcy .... #f29.status = post_beta #f29.post_beta.mandatory_days_in_testing = 7 #f29.post_beta.critpath.min_karma = 2 #f29.post_beta.critpath.stable_after_days_without_negative_karma = 14 #f29.status = pre_beta #f29.pre_beta.mandatory_days_in_testing = 3 #f29.pre_beta.critpath.min_karma = 1 #f29.pre_beta.critpath.stable_after_days_without_negative_karma = 14 .... == Configuring all bodhi nodes Run this command from the _ansible_ checkout to configure all of bodhi in production: .... # This will configure the backends $ sudo rbac-playbook playbooks/groups/bodhi2.yml # This will configure the frontend $ sudo rbac-playbook openshift-apps/bodhi.yml .... == Pushing updates SSH into the `bodhi-backend01` machine and run: .... $ sudo -u apache bodhi-push .... You can restrict the updates by release and/or request: .... $ sudo -u apache bodhi-push --releases f23,f22 --request stable .... You can also push specific builds: .... $ sudo -u apache bodhi-push --builds openssl-1.0.1k-14.fc22,openssl-1.0.1k-14.fc23 .... This will display a list of updates that are ready to be pushed. == Monitoring the bodhi composer output You can monitor the bodhi composer via the `bodhi` CLI tool, or via the systemd journal on `bodhi-backend01`: .... # From the comfort of your own laptop. $ bodhi composes list # From bodhi-backend01 $ journalctl -f -u fedmsg-hub .... == Resuming a failed push If a push fails for some reason, you can easily resume it on `bodhi-backend01` by running: .... $ sudo -u apache bodhi-push --resume .... == Release EOL .... bodhi-manage-releases edit --name F21 --state archived .... == Adding notices to the front page or new update form You can easily add notification messages to the front page of bodhi using the _frontpage_notice_ option in _ansible/roles/bodhi2/base/templates/production.ini.j2_. If you want to flash a message on the New Update Form, you can use the _newupdate_notice_ variable instead. This can be useful for announcing things like service outages, etc. == Using the Bodhi Shell to modify updates by hand The "bodhi shell" is a Python shell with the SQLAlchemy session and transaction manager initialized. It can be run from any production/staging backend instance and allows you to modify any models by hand. .... sudo pshell /etc/bodhi/production.ini # Execute a script that sets up the `db` and provides a `delete_update` function. # This will eventually be shipped in the bodhi package, but can also be found here. # https://raw.githubusercontent.com/fedora-infra/bodhi/develop/tools/shelldb.py >>> execfile('shelldb.py') .... At this point you have access to a _db_ SQLAlchemy Session instance, a _t_ _transaction module_, and _m_ for the _bodhi.models_. .... # Fetch an update, and tweak it as necessary. >>> up = m.Update.get(u'u'FEDORA-2016-4d226a5f7e', db) # Commit the transaction >>> t.commit() .... Here is an example of merging two updates together and deleting the original. .... >>> up = m.Update.get(u'FEDORA-2016-4d226a5f7e', db) >>> up.builds [, ] >>> b = up.builds[0] >>> up2 = m.Update.get(u'FEDORA-2016-5f63a874ca', db) >>> up2.builds [] >>> up.builds.remove(b) >>> up.builds.append(up2.builds[0]) >>> delete_update(up2) >>> t.commit() .... == Using the Bodhi shell to fix uniqueness problems with e-mail addresses Bodhi currently enforces uniqueness on user e-mail addresses. There is https://github.com/fedora-infra/bodhi/issues/2387[an issue] filed to drop this upstream, but for the time being the constraint is enforced. This can be a problem for users who have more than one FAS account if they make one account use an e-mail address that was previously used by another account, if that other account has not logged into Bodhi since it was changed to use a different address. One way the user can fix this themselves is to log in to Bodhi with the old account so that Bodhi learns about its new address. However, an admin can also fix this by hand by using the Bodhi shell. For example, suppose a user has created `user_1` and `user_2`. Suppose that `user_1` used to use `email_a@example.com` but has been changed to use `email_b@example.com` in FAS, and `user_2` is now configured to use `email_a@example.com` in FAS. If `user_2` attempts to log in to Bodhi, it will cause a uniqueness violation since Bodhi does not know that `user_1` has changed to `email_b@example.com`. The user can simply log in as `user_1` to fix this, which will cause Bodhi to update its e-mail address to `email_b@example.com`. Or an admin can fix it with a shell on one of the Bodhi backend servers like this: .... [bowlofeggs@bodhi-backend02 ~][PROD]$ sudo -u apache pshell /etc/bodhi/production.ini 2018-05-29 20:21:36,366 INFO [bodhi][MainThread] Using python-bugzilla 2018-05-29 20:21:36,367 DEBUG [bodhi][MainThread] Using Koji Buildsystem 2018-05-29 20:21:42,559 INFO [bodhi.server][MainThread] Bodhi ready and at your service! Python 2.7.14 (default, Mar 14 2018, 13:36:31) [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2 Type "help" for more information. Environment: app The WSGI application. registry Active Pyramid registry. request Active request object. root Root of the default resource tree. root_factory Default root factory used to create `root`. Custom Variables: m bodhi.server.models >>> u = m.User.query.filter_by(name=u'user_1').one() >>> u.email = u'email_b@example.com' >>> m.Session().commit() .... == Troubleshooting and Resolution === Atomic OSTree compose failure If the Atomic OSTree compose fails with some sort of _Device or Resource busy_ error, then run _mount_ to see if there are any stray _tmpfs_ mounts still active: .... tmpfs on /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq type tmpfs (rw,relatime,seclabel,mode=755) .... You can then _umount /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq_ and resume the push again. === nfs repodata cache IOError Sometimes you may hit an IOError during the updateinfo.xml generation process from createrepo_c: .... IOError: Cannot open /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml: File /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml doesn't exists or not a regular file .... This issue will be resolved with NFSv4, but in the mean time it can be worked around by removing the _.repocache_ directory and resuming the push: .... rm -fr /mnt/koji/mash/updates/epel7.repocache ....