diff --git a/modules/sysadmin_guide/pages/koji-builder-setup.adoc b/modules/sysadmin_guide/pages/koji-builder-setup.adoc index 81c5a5e..ab4038e 100644 --- a/modules/sysadmin_guide/pages/koji-builder-setup.adoc +++ b/modules/sysadmin_guide/pages/koji-builder-setup.adoc @@ -15,13 +15,24 @@ It is recommended to upgrade or reinstall all builders after each release. To do Follow the process to upgrade builders in staging environment: . Update the virt-install path on the builders in ansible repository, like in this https://pagure.io/fedora-infra/ansible/c/af94db8ab88588aa8dc2c20893b2487afb7f7777?branch=main[example] -. Each builder virtual machine is removed, either via virt-inst-dest or manually, and the buildvm.yml playbook is run to deploy them. -. The hub is re-installed or upgraded with 'dnf --refresh --releasever distro-sync'. +. Each builder virtual machine is removed, either via `virt-inst-dest` or manually, and the `buildvm.yml` playbook is run to deploy them. +. The hub is re-installed or upgraded with `dnf --refresh --releasever distro-sync`. . Test builds are done to make sure everything is working. === Production Environment -For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. +For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From _buildvm-s390x-01_ to _buildvm-s390x-14_ are **zvm** instances, they have to be updated with `dnf --releasever N distro-sync`. +In production builders, are usually done per bvmhost at a time: + +. To do this process the user should be koji admin +. Find all the builders on the host looking in `/var/log/virthost-lists.out` on batcave01 or look for `vmhost` variable in `inventory/host_vars/` on https://pagure.io/fedora-infra/ansible[ansible repository] +. In a loop, disable all those builders using `koji disable-host ` +. In a loop check for running builds on them with `koji list-tasks --host ` +. Either wait for all of them to finish or optionally run `koji free-task NNNNN` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers +. Once they are all empty, destroy them as above and reinstall them with the `buildvm` playbook. Optionally, this is a good time to update the bvmhost and reboot it after destroying, but before installing builders +. After installed, check that each builder is checking in with `koji list-hosts | grep ` +. In a loop, reenable them + ////