Infrastructure/infra-docs-fpo
1
0
Fork 0
Code Issues 100 Pull requests 6 Projects Releases Packages Wiki Activity Actions

added production section

Browse source 
Signed-off-by: Pedro Moura <pmoura@redhat.com>
This commit is contained in:
Pedro Moura 2022-03-16 18:16:22 -03:00 • committed by kevin
parent 42d42428cf
commit b8a08cdfb5
1 changed files with 14 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

17
modules/sysadmin_guide/pages/koji-builder-setup.adoc
View file

@ -15,13 +15,24 @@ It is recommended to upgrade or reinstall all builders after each release. To do
Follow the process to upgrade builders in staging environment:
. Update the virt-install path on the builders in ansible repository, like in this https://pagure.io/fedora-infra/ansible/c/af94db8ab88588aa8dc2c20893b2487afb7f7777?branch=main[example]
. Each builder virtual machine is removed, either via virt-inst-dest or manually, and the buildvm.yml playbook is run to deploy them.
. The hub is re-installed or upgraded with 'dnf --refresh --releasever <release> distro-sync'.
. Each builder virtual machine is removed, either via `virt-inst-dest` or manually, and the `buildvm.yml` playbook is run to deploy them.
. The hub is re-installed or upgraded with `dnf --refresh --releasever <release> distro-sync`.
. Test builds are done to make sure everything is working.
=== Production Environment
For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time.
For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From _buildvm-s390x-01_ to _buildvm-s390x-14_ are **zvm** instances, they have to be updated with `dnf --releasever N distro-sync`.
In production builders, are usually done per bvmhost at a time:
. To do this process the user should be koji admin
. Find all the builders on the host looking in `/var/log/virthost-lists.out` on batcave01 or look for `vmhost` variable in `inventory/host_vars/` on https://pagure.io/fedora-infra/ansible[ansible repository]
. In a loop, disable all those builders using `koji disable-host <vmname>`
. In a loop check for running builds on them with `koji list-tasks --host <vmname>`
. Either wait for all of them to finish or optionally run `koji free-task NNNNN` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers
. Once they are all empty, destroy them as above and reinstall them with the `buildvm` playbook. Optionally, this is a good time to update the bvmhost and reboot it after destroying, but before installing builders
. After installed, check that each builder is checking in with `koji list-hosts | grep <vmname>`
. In a loop, reenable them
////