Split the bodhi SOP to dedployment and releng one
This commit is contained in:
Adam Saleh
parent
0158707f1e
commit
6fc255a411
3 changed files with 281 additions and 213 deletions
|
|
@ -10,7 +10,8 @@
|
||||||
** xref:aws-access.adoc[Amazon Web Services Access - SOP]
|
** xref:aws-access.adoc[Amazon Web Services Access - SOP]
|
||||||
** xref:bastion-hosts-info.adoc[Fedora Bastion Hosts - SOP]
|
** xref:bastion-hosts-info.adoc[Fedora Bastion Hosts - SOP]
|
||||||
** xref:blockerbugs.adoc[Blockerbugs Infrastructure - SOP]
|
** xref:blockerbugs.adoc[Blockerbugs Infrastructure - SOP]
|
||||||
** xref:bodhi.adoc[Bodhi Infrastructure - SOP]
|
** xref:bodhi.adoc[Bodhi Infrastructure - Releng SOP]
|
||||||
|
** xref:bodhi-deploy.adoc[Bodhi Infrastructure - Deployment SOP]
|
||||||
** xref:bugzilla.adoc[Bugzilla Sync Infrastructure - SOP]
|
** xref:bugzilla.adoc[Bugzilla Sync Infrastructure - SOP]
|
||||||
** xref:bugzilla2fedmsg.adoc[bugzilla2fedmsg - SOP]
|
** xref:bugzilla2fedmsg.adoc[bugzilla2fedmsg - SOP]
|
||||||
** xref:collectd.adoc[Collectd - SOP]
|
** xref:collectd.adoc[Collectd - SOP]
|
||||||
|
|
|
||||||
278
modules/sysadmin_guide/pages/bodhi-deploy.adoc
Normal file
278
modules/sysadmin_guide/pages/bodhi-deploy.adoc
Normal file
|
|
@ -0,0 +1,278 @@
|
||||||
|
= Bodhi Infrastructure Release and Deployment SOP
|
||||||
|
|
||||||
|
Bodhi is used by Fedora developers to submit potential package updates
|
||||||
|
for releases and to manage buildroot overrides. From here, bodhi handles
|
||||||
|
all of the dirty work, from sending around emails, dealing with Koji, to
|
||||||
|
composing the repositories.
|
||||||
|
|
||||||
|
Bodhi production instance: https://bodhi.fedoraproject.org
|
||||||
|
|
||||||
|
Bodhi project page: https://github.com/fedora-infra/bodhi
|
||||||
|
|
||||||
|
=== Necessary permissions
|
||||||
|
|
||||||
|
Groups to be member in
|
||||||
|
|
||||||
|
* https://github.com/orgs/fedora-infra/teams/bodhi
|
||||||
|
* https://src.fedoraproject.org/group/infra-sig
|
||||||
|
* https://accounts.fedoraproject.org/group/infra-sig/
|
||||||
|
* https://accounts.fedoraproject.org/group/packager/
|
||||||
|
* https://accounts.fedoraproject.org/group/sysadmin/
|
||||||
|
* https://accounts.fedoraproject.org/group/sysadmin-noc/
|
||||||
|
* https://accounts.fedoraproject.org/group/sysadmin-bodhi/
|
||||||
|
|
||||||
|
In addition you should be named as app-owner in the bodhi os playbook.
|
||||||
|
|
||||||
|
== Performing a bodhi upgrade
|
||||||
|
|
||||||
|
=== Have the release in the official Bodhi repository ready
|
||||||
|
|
||||||
|
Following steps were taken from
|
||||||
|
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst_]
|
||||||
|
up to the point of building a beta-release.
|
||||||
|
|
||||||
|
Changing the specfile manually is tedious and error-prone, it is more
|
||||||
|
useful to just do several patch releases on staging, before we commit to
|
||||||
|
one to deploy on production.
|
||||||
|
|
||||||
|
==== Releasing a patch-release
|
||||||
|
|
||||||
|
Make sure you have all of the code you want to release in the branch. I.e. for 5.7.3 release,
|
||||||
|
you would be looking at https://github.com/fedora-infra/bodhi/tree/5.7
|
||||||
|
|
||||||
|
Easiest way to incorporate additional changes from PRs targetting develop,
|
||||||
|
is to tag them with `*-backports` tag, i.e. `5.7-backports`. Mergify will automatically create a cherry-picked PR
|
||||||
|
targetting the release branch.
|
||||||
|
|
||||||
|
When you are ready to denote the new patch version:
|
||||||
|
|
||||||
|
1. bump __version__ in ./bodhi/__init__.py
|
||||||
|
2. run the get-authors.py in ./news
|
||||||
|
3. run towncrier
|
||||||
|
4. review the docs/user/release_notes.rst
|
||||||
|
5. commit, push to your branch and make a PR against develop with the corect `*-backports` label
|
||||||
|
|
||||||
|
After all of the tests successfully passed in the PRs, and both of them are merged, tag the release:
|
||||||
|
1. git checkout 5.7
|
||||||
|
2. git pull origin 5.7
|
||||||
|
3. git tag 5.7.3
|
||||||
|
4. git push origin refs/tags/5.7.3
|
||||||
|
|
||||||
|
If you push the wrong tag, you can delete it by `git push origin :refs/tags/5.7.3`
|
||||||
|
|
||||||
|
Beware, mergify creates the bacport PR only after the pr to develop was merged.
|
||||||
|
|
||||||
|
==== Testing deployed Bodhi without making a release
|
||||||
|
|
||||||
|
If the Bodhi changes don’t involve any of
|
||||||
|
|
||||||
|
* Composer code
|
||||||
|
* Doesn’t include any upgrades to database with alembic
|
||||||
|
|
||||||
|
You can opt to use the hotfix-style container-image that can be found
|
||||||
|
among the templates:
|
||||||
|
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix_]
|
||||||
|
|
||||||
|
Change the env-vars in the docker-file-base-hotfix to reflect the commit
|
||||||
|
to be built. The image definition is not buildable locally, as it needs
|
||||||
|
the openshift environment to configure koji and kerberos.
|
||||||
|
|
||||||
|
Change the buildconfig to point to the new definition:
|
||||||
|
|
||||||
|
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14_]
|
||||||
|
|
||||||
|
Then commit and push the changes, to be able to deploy from batcave
|
||||||
|
with:
|
||||||
|
|
||||||
|
sudo rbac-playbook openshift-apps/bodhi.yml -l staging
|
||||||
|
|
||||||
|
Make sure to check os.stg.fedoraproject.org to see if the build completed successfully.
|
||||||
|
|
||||||
|
==== Build Bodhi
|
||||||
|
|
||||||
|
Bodhi is deployed from the infrastructure Koji repositories. At the time
|
||||||
|
of this writing, it is deployed from the f34-infra and f34-infra-stg
|
||||||
|
(for staging) repositories. Bodhi is built for these repositories from
|
||||||
|
the master branch of thehttps://src.fedoraproject.org/rpms/bodhi[
|
||||||
|
]https://src.fedoraproject.org/rpms/bodhi[_bodhi dist-git repository_].
|
||||||
|
|
||||||
|
You can use fedpkg to download the package:
|
||||||
|
....
|
||||||
|
fedpkg clone rpms/bodhi
|
||||||
|
....
|
||||||
|
|
||||||
|
Curently we are interested in f34 builds as all of the bodhi infra runs on f34 servers:
|
||||||
|
|
||||||
|
....
|
||||||
|
fedpkg switch-branch f34
|
||||||
|
....
|
||||||
|
|
||||||
|
Edit the bodhi.spec specfile to have the correct release version.
|
||||||
|
|
||||||
|
Download the sources to the SOURCES folder (if it doesn't exist, just mkdir -p /rpmbuild/SOURCES/)
|
||||||
|
|
||||||
|
....
|
||||||
|
wget https://github.com/fedora-infra/bodhi/archive/refs/tags/5.7.3.tar.gz -o /rpmbuild/SOURCES/bodhi-5.7.3.tar.gz
|
||||||
|
....
|
||||||
|
|
||||||
|
As an example, to build a Bodhi for the f34-infra-stg repository, you
|
||||||
|
can use this command:
|
||||||
|
|
||||||
|
....
|
||||||
|
$ rpmbuild --define "dist .f34.infra" -bs bodhi.spec
|
||||||
|
....
|
||||||
|
|
||||||
|
Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
|
||||||
|
|
||||||
|
....
|
||||||
|
$ koji build f34-infra ~/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
|
||||||
|
....
|
||||||
|
|
||||||
|
When building a Bodhi release that is intended for production, we should
|
||||||
|
build from the production dist-git repo instead of uploading an SRPM:
|
||||||
|
|
||||||
|
$ koji build f34-infra
|
||||||
|
git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37
|
||||||
|
|
||||||
|
All builds against the f34-infra build target will go into the
|
||||||
|
f34-infra-stg repository. If you wish to promote a build from staging to
|
||||||
|
production, you can do something like this command:
|
||||||
|
|
||||||
|
$ koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.f34.infra
|
||||||
|
|
||||||
|
=== Staging
|
||||||
|
|
||||||
|
The upgrade playbook will apply configuration changes after running the
|
||||||
|
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
||||||
|
systems in order to get the upgrade playbook to succeed. If you are in
|
||||||
|
this situation, you can apply those changes by running the bodhi-backend
|
||||||
|
playbook:
|
||||||
|
|
||||||
|
....
|
||||||
|
sudo rbac-playbook -l staging groups/bodhi-backend.yml
|
||||||
|
....
|
||||||
|
|
||||||
|
In
|
||||||
|
the https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg, edit the bodhi_version setting to the version you wish to
|
||||||
|
deploy to staging. For example, to deploy version 5.7.3 to
|
||||||
|
staging, I would set that varible like this:
|
||||||
|
|
||||||
|
bodhi_version: ‘5.7.3’
|
||||||
|
|
||||||
|
The package itself will then be installed in container build by ‘dnf
|
||||||
|
install -y bodhi-server-\{\{bodhi_version}}’
|
||||||
|
|
||||||
|
Beware, when running the playbooks on the backend vm, it currently
|
||||||
|
installs the latest bodhi package available in the respective
|
||||||
|
infra/infra-stg tag.
|
||||||
|
|
||||||
|
It can be useful to verify correct version is available on the backend,
|
||||||
|
i.e. for staging run
|
||||||
|
|
||||||
|
....
|
||||||
|
ssh bodhi-backend01.stg.iad2.fedoraproject.org
|
||||||
|
....
|
||||||
|
|
||||||
|
....
|
||||||
|
dnf list bodhi-server --refresh --all --showduplicates
|
||||||
|
....
|
||||||
|
|
||||||
|
In future we will probably pin these in the VM as well.
|
||||||
|
|
||||||
|
Make sure you have the correct docker-image set in the configuration.
|
||||||
|
in the ansible repo:
|
||||||
|
....
|
||||||
|
vim roles/openshift-apps/bodhi/templates/buildconfig.yml
|
||||||
|
....
|
||||||
|
|
||||||
|
After you checked, and have the correct version, commit and create a PR for ansible repo.
|
||||||
|
|
||||||
|
After it has been merged, run these commands:
|
||||||
|
|
||||||
|
....
|
||||||
|
# Synchronize the database from production to staging
|
||||||
|
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
|
||||||
|
# Upgrade the Bodhi backend on staging
|
||||||
|
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging
|
||||||
|
# Upgrade the Bodhi frontend on staging
|
||||||
|
$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging
|
||||||
|
....
|
||||||
|
|
||||||
|
==== Tests to run
|
||||||
|
|
||||||
|
For compose test you can follow
|
||||||
|
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta_]
|
||||||
|
|
||||||
|
For automated health-check, you can run
|
||||||
|
https://pagure.io/fedora-ci/monitor-gating[_https://pagure.io/fedora-ci/monitor-gating_],
|
||||||
|
|
||||||
|
It should be sufficient to check the openshift project
|
||||||
|
https://os.fedoraproject.org/console/project/monitor-gating[_https://os.fedoraproject.org/console/project/monitor-gating_]
|
||||||
|
where it is being run every few hours. There is a dashboard available
|
||||||
|
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/6q2qpcvGz/monitor-gating
|
||||||
|
|
||||||
|
==== Production
|
||||||
|
|
||||||
|
After testing in staging, you should be able to move the rpm from
|
||||||
|
staging to production:
|
||||||
|
|
||||||
|
....
|
||||||
|
koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.fc34
|
||||||
|
....
|
||||||
|
|
||||||
|
Even if the rpm version is pinned in your deploy, double check the
|
||||||
|
availability in the tag
|
||||||
|
|
||||||
|
The upgrade playbook will apply configuration changes after running the
|
||||||
|
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
||||||
|
systems in order to get the upgrade playbook to succeed. If you are in
|
||||||
|
this situation, you can apply those changes by running the bodhi-backend
|
||||||
|
playbook:
|
||||||
|
|
||||||
|
....
|
||||||
|
sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend
|
||||||
|
....
|
||||||
|
|
||||||
|
In
|
||||||
|
thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[
|
||||||
|
]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[_os_masters
|
||||||
|
inventory_], edit the bodhi_version setting to the version you wish to
|
||||||
|
deploy to production. For example, to deploy bodhi-5.7.3-0.f34.infra to
|
||||||
|
production, I would set that varible like this:
|
||||||
|
|
||||||
|
....
|
||||||
|
bodhi_version: "5.7.3"
|
||||||
|
....
|
||||||
|
|
||||||
|
To update the bodhi RPMs in production:
|
||||||
|
|
||||||
|
....
|
||||||
|
# Update the backend VMs (this will also run the migrations, if any)
|
||||||
|
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend
|
||||||
|
# Update the frontend
|
||||||
|
$ sudo rbac-playbook openshift-apps/bodhi.yml
|
||||||
|
....
|
||||||
|
|
||||||
|
All of the playbooks should finish with no failures.
|
||||||
|
|
||||||
|
Afterwards, check
|
||||||
|
https://os.fedoraproject.org/console/project/bodhi/overview[_https://os.fedoraproject.org/console/project/bodhi/overview_]
|
||||||
|
to see if the image has been built and if it has been properly deployed.
|
||||||
|
|
||||||
|
Things to watchout for
|
||||||
|
|
||||||
|
* Correct rpm not being available for container build
|
||||||
|
** This is the reason to check the infra-tag for bodhi
|
||||||
|
* Failure to start the pod due to backend db mismatch
|
||||||
|
** Sometimes the alembic upgrade that is supposed to run as part of
|
||||||
|
playbooks/manual/upgrade/bodhi.yml fails silently
|
||||||
|
* Some issues resurface only after few days in production, resulting in
|
||||||
|
visible stack-traces in logs.
|
||||||
|
|
||||||
|
=== Syncing the production database to staging
|
||||||
|
|
||||||
|
This can be useful for testing issues with production data in staging:
|
||||||
|
|
||||||
|
....
|
||||||
|
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
|
||||||
|
....
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
= Bodhi Infrastructure SOP
|
= Bodhi Infrastructure Releng SOP
|
||||||
|
|
||||||
Bodhi is used by Fedora developers to submit potential package updates
|
Bodhi is used by Fedora developers to submit potential package updates
|
||||||
for releases and to manage buildroot overrides. From here, bodhi handles
|
for releases and to manage buildroot overrides. From here, bodhi handles
|
||||||
|
|
@ -190,217 +190,6 @@ If a push fails for some reason, you can easily resume it on
|
||||||
$ sudo -u apache bodhi-push --resume
|
$ sudo -u apache bodhi-push --resume
|
||||||
....
|
....
|
||||||
|
|
||||||
== Performing a bodhi upgrade
|
|
||||||
|
|
||||||
==== Have the release in the official Bodhi repository ready
|
|
||||||
|
|
||||||
Follow steps in
|
|
||||||
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst_]
|
|
||||||
up to the point of building a beta-release.
|
|
||||||
|
|
||||||
Changing the specfile manually is tedious and error-prone, it is more
|
|
||||||
useful to just do several patch releases on staging, before we commit to
|
|
||||||
one to deploy on production.
|
|
||||||
|
|
||||||
==== Testing deployed Bodhi without making a release
|
|
||||||
|
|
||||||
If the Bodhi changes don’t involve any of
|
|
||||||
|
|
||||||
* Composer code
|
|
||||||
* Doesn’t include any upgrades to database with alembic
|
|
||||||
|
|
||||||
You can opt to use the hotfix-style container-image that can be found
|
|
||||||
among the templates:
|
|
||||||
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix_]
|
|
||||||
|
|
||||||
Change the env-vars in the docker-file-base-hotfix to reflect the commit
|
|
||||||
to be built. The image definition is not buildable locally, as it needs
|
|
||||||
the openshift environment to configure koji and kerberos.
|
|
||||||
|
|
||||||
Change the buildconfig to point to the new definition:
|
|
||||||
|
|
||||||
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14_]
|
|
||||||
|
|
||||||
I.e:
|
|
||||||
|
|
||||||
[cols=",",]
|
|
||||||
|===
|
|
||||||
| |\{% if env == "staging" %}
|
|
||||||
| | \{\{ load_file('dockerfile-base-hotfix') | indent(8) }}
|
|
||||||
| |\{% else %}
|
|
||||||
| | \{\{ load_file('dockerfile-base') | indent(8) }}
|
|
||||||
| |\{% endif %}
|
|
||||||
|===
|
|
||||||
|
|
||||||
Then commit and push the changes, to be able to deploy from batcave
|
|
||||||
with:
|
|
||||||
|
|
||||||
sudo rbac-playbook openshift-apps/bodhi.yml -l staging
|
|
||||||
|
|
||||||
==== Build Bodhi
|
|
||||||
|
|
||||||
Bodhi is deployed from the infrastructure Koji repositories. At the time
|
|
||||||
of this writing, it is deployed from the f34-infra and f34-infra-stg
|
|
||||||
(for staging) repositories. Bodhi is built for these repositories from
|
|
||||||
the master branch of thehttps://src.fedoraproject.org/rpms/bodhi[
|
|
||||||
]https://src.fedoraproject.org/rpms/bodhi[_bodhi dist-git repository_].
|
|
||||||
|
|
||||||
As an example, to build a Bodhi for the f34-infra-stg repository, you
|
|
||||||
can use this command:
|
|
||||||
|
|
||||||
....
|
|
||||||
$ rpmbuild --define "dist .f34.infra" -bs bodhi.spec
|
|
||||||
....
|
|
||||||
|
|
||||||
Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
|
|
||||||
|
|
||||||
....
|
|
||||||
$ koji build f34-infra
|
|
||||||
/home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
|
|
||||||
....
|
|
||||||
|
|
||||||
When building a Bodhi release that is intended for production, we should
|
|
||||||
build from the production dist-git repo instead of uploading an SRPM:
|
|
||||||
|
|
||||||
$ koji build f34-infra
|
|
||||||
git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37
|
|
||||||
|
|
||||||
All builds against the f34-infra build target will go into the
|
|
||||||
f34-infra-stg repository. If you wish to promote a build from staging to
|
|
||||||
production, you can do something like this command:
|
|
||||||
|
|
||||||
$ koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.f34.infra
|
|
||||||
|
|
||||||
=== Staging
|
|
||||||
|
|
||||||
The upgrade playbook will apply configuration changes after running the
|
|
||||||
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
|
||||||
systems in order to get the upgrade playbook to succeed. If you are in
|
|
||||||
this situation, you can apply those changes by running the bodhi-backend
|
|
||||||
playbook:
|
|
||||||
|
|
||||||
....
|
|
||||||
sudo rbac-playbook -l staging groups/bodhi-backend.yml
|
|
||||||
....
|
|
||||||
|
|
||||||
In
|
|
||||||
thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[
|
|
||||||
]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[_os_masters
|
|
||||||
inventory_], edit the bodhi_version setting to the version you wish to
|
|
||||||
deploy to staging. For example, to deploy version 5.7.3 to
|
|
||||||
staging, I would set that varible like this:
|
|
||||||
|
|
||||||
bodhi_version: ‘5.7.3’
|
|
||||||
|
|
||||||
The package itself will then be installed in container build by ‘dnf
|
|
||||||
install -y bodhi-server-\{\{bodhi_version}}’
|
|
||||||
|
|
||||||
Beware, when running the playbooks on the backend vm, it currently
|
|
||||||
installs the latest bodhi package available in the respective
|
|
||||||
infra/infra-stg tag.
|
|
||||||
|
|
||||||
It can be useful to verify correct version is available on the backend,
|
|
||||||
i.e. for staging run
|
|
||||||
|
|
||||||
....
|
|
||||||
ssh bodhi-backend01.stg.iad2.fedoraproject.org
|
|
||||||
....
|
|
||||||
|
|
||||||
....
|
|
||||||
dnf list bodhi-server --refresh --all --showduplicates
|
|
||||||
....
|
|
||||||
|
|
||||||
*MAYBE WE SHOULD PIN THIS AS WELL!!!*
|
|
||||||
|
|
||||||
Run these commands:
|
|
||||||
|
|
||||||
....
|
|
||||||
# Synchronize the database from production to staging
|
|
||||||
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
|
|
||||||
# Upgrade the Bodhi backend on staging
|
|
||||||
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging
|
|
||||||
# Upgrade the Bodhi frontend on staging
|
|
||||||
$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging
|
|
||||||
....
|
|
||||||
|
|
||||||
==== Tests to run
|
|
||||||
|
|
||||||
For compose test you can follow
|
|
||||||
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta_]
|
|
||||||
|
|
||||||
For automated health-check, you can run
|
|
||||||
https://pagure.io/fedora-ci/monitor-gating[_https://pagure.io/fedora-ci/monitor-gating_],
|
|
||||||
|
|
||||||
It should be sufficient to check the openshift project
|
|
||||||
https://os.fedoraproject.org/console/project/monitor-gating[_https://os.fedoraproject.org/console/project/monitor-gating_]
|
|
||||||
where it is being run every few hours. There is a dashboard available
|
|
||||||
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/6q2qpcvGz/monitor-gating
|
|
||||||
|
|
||||||
==== Production
|
|
||||||
|
|
||||||
After testing in staging, you should be able to move the rpm from
|
|
||||||
staging to production:
|
|
||||||
|
|
||||||
....
|
|
||||||
koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.fc34
|
|
||||||
....
|
|
||||||
|
|
||||||
Even if the rpm version is pinned in your deploy, double check the
|
|
||||||
availability in the tag
|
|
||||||
|
|
||||||
The upgrade playbook will apply configuration changes after running the
|
|
||||||
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
|
||||||
systems in order to get the upgrade playbook to succeed. If you are in
|
|
||||||
this situation, you can apply those changes by running the bodhi-backend
|
|
||||||
playbook:
|
|
||||||
|
|
||||||
....
|
|
||||||
sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend
|
|
||||||
....
|
|
||||||
|
|
||||||
In
|
|
||||||
thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[
|
|
||||||
]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[_os_masters
|
|
||||||
inventory_], edit the bodhi_version setting to the version you wish to
|
|
||||||
deploy to production. For example, to deploy bodhi-5.7.3-0.f34.infra to
|
|
||||||
production, I would set that varible like this:
|
|
||||||
|
|
||||||
....
|
|
||||||
bodhi_version: "5.7.3"
|
|
||||||
....
|
|
||||||
|
|
||||||
To update the bodhi RPMs in production:
|
|
||||||
|
|
||||||
....
|
|
||||||
# Update the backend VMs (this will also run the migrations, if any)
|
|
||||||
$ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend
|
|
||||||
# Update the frontend
|
|
||||||
$ sudo rbac-playbook openshift-apps/bodhi.yml
|
|
||||||
....
|
|
||||||
|
|
||||||
All of the playbooks should finish with no failures.
|
|
||||||
|
|
||||||
Afterwards, check
|
|
||||||
https://os.fedoraproject.org/console/project/bodhi/overview[_https://os.fedoraproject.org/console/project/bodhi/overview_]
|
|
||||||
to see if the image has been built and if it has been properly deployed.
|
|
||||||
|
|
||||||
Things to watchout for
|
|
||||||
|
|
||||||
* Correct rpm not being available for container build
|
|
||||||
** This is the reason to check the infra-tag for bodhi
|
|
||||||
* Failure to start the pod due to backend db mismatch
|
|
||||||
** Sometimes the alembic upgrade that is supposed to run as part of
|
|
||||||
playbooks/manual/upgrade/bodhi.yml fails silently
|
|
||||||
* Some issues resurface only after few days in production, resulting in
|
|
||||||
visible stack-traces in logs.
|
|
||||||
|
|
||||||
=== Syncing the production database to staging
|
|
||||||
|
|
||||||
This can be useful for testing issues with production data in staging:
|
|
||||||
|
|
||||||
....
|
|
||||||
$ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging
|
|
||||||
....
|
|
||||||
|
|
||||||
== Release EOL
|
== Release EOL
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue