It is recommended to upgrade or reinstall all builders after each release. To do it, usually the builders are upgraded and virtual machines reinstalled. First they should be upgraded in staging environment to make sure everything is working properly before starting on production.
. Update the virt-install path on the builders in ansible repository, like in this https://pagure.io/fedora-infra/ansible/c/af94db8ab88588aa8dc2c20893b2487afb7f7777?branch=main[example]
For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From _buildvm-s390x-01_ to _buildvm-s390x-14_ are **zvm** instances, they have to be updated with `dnf --releasever NAME distro-sync`.
In production builders, are usually done per bvmhost at a time:
. To do this process the user should be koji admin
. Find all the builders on the host looking in `/var/log/virthost-lists.out` on batcave01 or look for `vmhost` variable in `inventory/host_vars/` on https://pagure.io/fedora-infra/ansible[ansible repository]
. Either wait for all of them to finish or optionally run `koji free-task NAME` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers
. reinstall them with the `buildvm` playbook. Optionally, this is a good time to update the bvmhost and reboot it after destroying, but before installing builders
. disable builder with koji `disable-host <buildhwname>`
. In a loop check for running builds on them with `koji list-tasks --host <vmname>`
. Either wait for all of them to finish or optionally run `koji free-task NAME` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers
. update with `dnf --releasever NAME distro-sync`
. reboot
. confirm the buildhw is back up and checking into koji with `koji list-hosts | grep buildhw`