Updating bodhi guide
This commit is contained in:
Adam Saleh
parent
c050563e2b
commit
0158707f1e
1 changed files with 137 additions and 31 deletions
|
|
@ -192,37 +192,84 @@ $ sudo -u apache bodhi-push --resume
|
|||
|
||||
== Performing a bodhi upgrade
|
||||
|
||||
=== Build Bodhi
|
||||
==== Have the release in the official Bodhi repository ready
|
||||
|
||||
Follow steps in
|
||||
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst_]
|
||||
up to the point of building a beta-release.
|
||||
|
||||
Changing the specfile manually is tedious and error-prone, it is more
|
||||
useful to just do several patch releases on staging, before we commit to
|
||||
one to deploy on production.
|
||||
|
||||
==== Testing deployed Bodhi without making a release
|
||||
|
||||
If the Bodhi changes don’t involve any of
|
||||
|
||||
* Composer code
|
||||
* Doesn’t include any upgrades to database with alembic
|
||||
|
||||
You can opt to use the hotfix-style container-image that can be found
|
||||
among the templates:
|
||||
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/dockerfile-base-hotfix_]
|
||||
|
||||
Change the env-vars in the docker-file-base-hotfix to reflect the commit
|
||||
to be built. The image definition is not buildable locally, as it needs
|
||||
the openshift environment to configure koji and kerberos.
|
||||
|
||||
Change the buildconfig to point to the new definition:
|
||||
|
||||
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14[_https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/bodhi/templates/buildconfig.yml#_14_]
|
||||
|
||||
I.e:
|
||||
|
||||
[cols=",",]
|
||||
|===
|
||||
| |\{% if env == "staging" %}
|
||||
| | \{\{ load_file('dockerfile-base-hotfix') | indent(8) }}
|
||||
| |\{% else %}
|
||||
| | \{\{ load_file('dockerfile-base') | indent(8) }}
|
||||
| |\{% endif %}
|
||||
|===
|
||||
|
||||
Then commit and push the changes, to be able to deploy from batcave
|
||||
with:
|
||||
|
||||
sudo rbac-playbook openshift-apps/bodhi.yml -l staging
|
||||
|
||||
==== Build Bodhi
|
||||
|
||||
Bodhi is deployed from the infrastructure Koji repositories. At the time
|
||||
of this writing, it is deployed from the `f29-infra` and `f29-infra-stg`
|
||||
of this writing, it is deployed from the f34-infra and f34-infra-stg
|
||||
(for staging) repositories. Bodhi is built for these repositories from
|
||||
the `master` branch of the
|
||||
https://src.fedoraproject.org/rpms/bodhi[bodhi dist-git repository].
|
||||
the master branch of thehttps://src.fedoraproject.org/rpms/bodhi[
|
||||
]https://src.fedoraproject.org/rpms/bodhi[_bodhi dist-git repository_].
|
||||
|
||||
As an example, to build a Bodhi beta for the `f29-infra-stg` repository,
|
||||
you can use this command:
|
||||
As an example, to build a Bodhi for the f34-infra-stg repository, you
|
||||
can use this command:
|
||||
|
||||
....
|
||||
$ rpmbuild --define "dist .fc29.infra" -bs bodhi.spec
|
||||
Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-3.13.0-0.0.beta.e0ca5bc.fc29.infra.src.rpm
|
||||
$ koji build f29-infra /home/bowlofeggs/rpmbuild/SRPMS/bodhi-3.13.0-0.0.beta.e0ca5bc.fc29.infra.src.rpm
|
||||
$ rpmbuild --define "dist .f34.infra" -bs bodhi.spec
|
||||
....
|
||||
|
||||
Wrote: /home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
|
||||
|
||||
....
|
||||
$ koji build f34-infra
|
||||
/home/bowlofeggs/rpmbuild/SRPMS/bodhi-5.7.3-0.f34.infra.src.rpm
|
||||
....
|
||||
|
||||
When building a Bodhi release that is intended for production, we should
|
||||
build from the production dist-git repo instead of uploading an SRPM:
|
||||
|
||||
....
|
||||
$ koji build f29-infra git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37
|
||||
....
|
||||
$ koji build f34-infra
|
||||
git+https://src.fedoraproject.org/rpms/bodhi.git#d64f40408876ec85663ec52888c4e44d92614b37
|
||||
|
||||
All builds against the `f29-infra` build target will go into the
|
||||
`f29-infra-stg` repository. If you wish to promote a build from staging
|
||||
to production, you can do something like this command:
|
||||
All builds against the f34-infra build target will go into the
|
||||
f34-infra-stg repository. If you wish to promote a build from staging to
|
||||
production, you can do something like this command:
|
||||
|
||||
....
|
||||
$ koji move-build f29-infra-stg f29-infra bodhi-3.13.0-1.fc29.infra
|
||||
....
|
||||
$ koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.f34.infra
|
||||
|
||||
=== Staging
|
||||
|
||||
|
|
@ -236,16 +283,35 @@ playbook:
|
|||
sudo rbac-playbook -l staging groups/bodhi-backend.yml
|
||||
....
|
||||
|
||||
In the
|
||||
https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[os_masters inventory],
|
||||
edit the `bodhi_version` setting to the version you wish to deploy to
|
||||
staging. For example, to deploy `bodhi-3.13.0-1.fc29.infra` to staging,
|
||||
I would set that varible like this:
|
||||
In
|
||||
thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[
|
||||
]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters_stg[_os_masters
|
||||
inventory_], edit the bodhi_version setting to the version you wish to
|
||||
deploy to staging. For example, to deploy version 5.7.3 to
|
||||
staging, I would set that varible like this:
|
||||
|
||||
bodhi_version: ‘5.7.3’
|
||||
|
||||
The package itself will then be installed in container build by ‘dnf
|
||||
install -y bodhi-server-\{\{bodhi_version}}’
|
||||
|
||||
Beware, when running the playbooks on the backend vm, it currently
|
||||
installs the latest bodhi package available in the respective
|
||||
infra/infra-stg tag.
|
||||
|
||||
It can be useful to verify correct version is available on the backend,
|
||||
i.e. for staging run
|
||||
|
||||
....
|
||||
bodhi_version: "bodhi-3.13.0-1.fc29.infra"
|
||||
ssh bodhi-backend01.stg.iad2.fedoraproject.org
|
||||
....
|
||||
|
||||
....
|
||||
dnf list bodhi-server --refresh --all --showduplicates
|
||||
....
|
||||
|
||||
*MAYBE WE SHOULD PIN THIS AS WELL!!!*
|
||||
|
||||
Run these commands:
|
||||
|
||||
....
|
||||
|
|
@ -257,7 +323,30 @@ $ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging
|
|||
$ sudo rbac-playbook openshift-apps/bodhi.yml -l staging
|
||||
....
|
||||
|
||||
=== Production
|
||||
==== Tests to run
|
||||
|
||||
For compose test you can follow
|
||||
https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta[_https://github.com/fedora-infra/bodhi/blob/develop/docs/developer/releases.rst#test-the-beta_]
|
||||
|
||||
For automated health-check, you can run
|
||||
https://pagure.io/fedora-ci/monitor-gating[_https://pagure.io/fedora-ci/monitor-gating_],
|
||||
|
||||
It should be sufficient to check the openshift project
|
||||
https://os.fedoraproject.org/console/project/monitor-gating[_https://os.fedoraproject.org/console/project/monitor-gating_]
|
||||
where it is being run every few hours. There is a dashboard available
|
||||
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/6q2qpcvGz/monitor-gating
|
||||
|
||||
==== Production
|
||||
|
||||
After testing in staging, you should be able to move the rpm from
|
||||
staging to production:
|
||||
|
||||
....
|
||||
koji move-build f34-infra-stg f34-infra bodhi-5.7.3-0.fc34
|
||||
....
|
||||
|
||||
Even if the rpm version is pinned in your deploy, double check the
|
||||
availability in the tag
|
||||
|
||||
The upgrade playbook will apply configuration changes after running the
|
||||
alembic upgrade. Sometimes you may need changes applied to the Bodhi
|
||||
|
|
@ -269,14 +358,15 @@ playbook:
|
|||
sudo rbac-playbook groups/bodhi-backend.yml -l bodhi-backend
|
||||
....
|
||||
|
||||
In the
|
||||
https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[os_masters inventory],
|
||||
edit the `bodhi_version` setting to the version you wish to deploy to
|
||||
production. For example, to deploy `bodhi-3.13.0-1.fc29.infra` to
|
||||
In
|
||||
thehttps://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[
|
||||
]https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/os_masters[_os_masters
|
||||
inventory_], edit the bodhi_version setting to the version you wish to
|
||||
deploy to production. For example, to deploy bodhi-5.7.3-0.f34.infra to
|
||||
production, I would set that varible like this:
|
||||
|
||||
....
|
||||
bodhi_version: "bodhi-3.13.0-1.fc29.infra"
|
||||
bodhi_version: "5.7.3"
|
||||
....
|
||||
|
||||
To update the bodhi RPMs in production:
|
||||
|
|
@ -288,7 +378,23 @@ $ sudo rbac-playbook manual/upgrade/bodhi.yml -l bodhi-backend
|
|||
$ sudo rbac-playbook openshift-apps/bodhi.yml
|
||||
....
|
||||
|
||||
== Syncing the production database to staging
|
||||
All of the playbooks should finish with no failures.
|
||||
|
||||
Afterwards, check
|
||||
https://os.fedoraproject.org/console/project/bodhi/overview[_https://os.fedoraproject.org/console/project/bodhi/overview_]
|
||||
to see if the image has been built and if it has been properly deployed.
|
||||
|
||||
Things to watchout for
|
||||
|
||||
* Correct rpm not being available for container build
|
||||
** This is the reason to check the infra-tag for bodhi
|
||||
* Failure to start the pod due to backend db mismatch
|
||||
** Sometimes the alembic upgrade that is supposed to run as part of
|
||||
playbooks/manual/upgrade/bodhi.yml fails silently
|
||||
* Some issues resurface only after few days in production, resulting in
|
||||
visible stack-traces in logs.
|
||||
|
||||
=== Syncing the production database to staging
|
||||
|
||||
This can be useful for testing issues with production data in staging:
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue