diff --git a/modules/sysadmin_guide/pages/koji-builder-setup.adoc b/modules/sysadmin_guide/pages/koji-builder-setup.adoc index ab4038e..adf183c 100644 --- a/modules/sysadmin_guide/pages/koji-builder-setup.adoc +++ b/modules/sysadmin_guide/pages/koji-builder-setup.adoc @@ -21,143 +21,22 @@ Follow the process to upgrade builders in staging environment: === Production Environment -For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From _buildvm-s390x-01_ to _buildvm-s390x-14_ are **zvm** instances, they have to be updated with `dnf --releasever N distro-sync`. +For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From _buildvm-s390x-01_ to _buildvm-s390x-14_ are **zvm** instances, they have to be updated with `dnf --releasever NAME distro-sync`. In production builders, are usually done per bvmhost at a time: . To do this process the user should be koji admin . Find all the builders on the host looking in `/var/log/virthost-lists.out` on batcave01 or look for `vmhost` variable in `inventory/host_vars/` on https://pagure.io/fedora-infra/ansible[ansible repository] -. In a loop, disable all those builders using `koji disable-host ` +. In a loop, disable all those builders using `koji disable-host `. Like the example below: ++ +.... +for i in seq -w 01 35 +do + echo $i + koji disable-host buildvm-s390x-$i.s390.fedoraproject.org +done +.... . In a loop check for running builds on them with `koji list-tasks --host ` -. Either wait for all of them to finish or optionally run `koji free-task NNNNN` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers +. Either wait for all of them to finish or optionally run `koji free-task NAME` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers . Once they are all empty, destroy them as above and reinstall them with the `buildvm` playbook. Optionally, this is a good time to update the bvmhost and reboot it after destroying, but before installing builders . After installed, check that each builder is checking in with `koji list-hosts | grep ` . In a loop, reenable them - - - -//// -From this comment below, everything is going to be deleted -Keeping all commented for reference while writing a new SOP -//// - -//// -= Setup Koji Builder SOP - -== Contents - -* <<_builder_setup>> -** <<_network_overview>> -** <<_setup_overview>> -** <<_resettinginstalling_an_old_koji_builder>> - -== Builder Setup - -Setting up a new koji builder involves a goodly number of steps: - -=== Network Overview - -[arabic] -. First get an instance spun up following the kickstart sop. -. Define a hostname for it on the 125 network and a $hostname-nfs name -for it on the .127 network. -. make sure the instance has 2 network connections: -* eth0 should be on the .125 network -* eth1 should be on the .127 network -+ -[NOTE] -===== -For VM _eth0_ should be on _br0_, _eth1_ on _br1_ on the vmhost. -===== - -=== Setup Overview - -* install the system as normal: -+ -.... -virt-install -n $builder_fqdn -r $memsize \ --f $path_to_lvm --vcpus=$numprocs \ --l http://10.5.126.23/repo/rhel/RHEL6-x86_64/ \ --x "ksdevice=eth0 ks=http://10.5.126.23/repo/rhel/ks/kvm-rhel-6 \ - ip=$ip netmask=$netmask gateway=$gw dns=$dns \ - console=tty0 console=ttyS0" \ - --network=bridge=br0 --network=bridge=br1 \ - --vnc --noautoconsole -.... -* run python `/root/tmp/setup-nfs-network.py` this should print out the --nfs hostname that you made above -* change root pw -* disable selinux on the machine in /etc/sysconfig/selinux -* reboot -* setup ssl cert into private/builders - use fqdn of host as DN -** login to _fas01_ as root -** `cd /var/lib/fedora-ca` -** `./kojicerthelper.py normal --outdir=/tmp/ \ --name=$fqdn_of_the_new_builder --cadir=. --caname=Fedora` -** info for the cert should be like this: -+ -.... -Country Name (2 letter code) [US]: -State or Province Name (full name) [North Carolina]: -Locality Name (eg, city) [Raleigh]: -Organization Name (eg, company) [Fedora Project]: -Organizational Unit Name (eg, section) []:Fedora Builders -Common Name (eg, your name or your servers hostname) []:$fqdn_of_new_builder -Email Address []:buildsys@fedoraproject.org -.... -** scp the file in `/tmp/$\{fqdn}_key_and_cert.pem` over to _batcave01_ -** put file in the private repo under `private/builders/$dn}.pem` -** `git add` + `git commit` -** `git push` -* run `./sync-hosts` in infra-hosts repo; `git commit; git push` -* as a koji admin run: -+ -.... -koji add-host $fqdnr i386 x86_64 - -(note: those are yum basearchs on the end - season to taste) -.... - -=== Resetting/installing an old koji builder - -* disable the builder in koji (ask a koji admin) -* halt the old system (`halt -p`) -* undefine the vm instance on the buildvmhost: -+ -.... -virsh undefine $builder_fqdn -.... -* reinstall it - from the buildvmhost run: -+ -.... -virt-install -n $builder_fqdn -r $memsize \ --f $path_to_lvm --vcpus=$numprocs \ - -l http://10.5.126.23/repo/rhel/RHEL6-x86_64/ \ - -x "ksdevice=eth0 ks=http://10.5.126.23/repo/rhel/ks/kvm-rhel-6 \ - ip=$ip netmask=$netmask gateway=$gw dns=$dns \ - console=tty0 console=ttyS0" \ - --network=bridge=br0 --network=bridge=br1 \ - --vnc --noautoconsole -.... -* watch install via vnc: -+ -.... -vncviewer -via bastion.fedoraproject.org $builder_fqdn:1 -.... -* when the install finishes: -** start the instance on the buildvmhost: -+ -.... -virsh start $builder_fqdn -.... -** set it to autostart on the buildvmhost: -+ -.... -virsh autostart $builder_fqdn -.... -* when the guest comes up -** login via ssh using the temp root password -** python `/root/tmp/setup-nfs-network.py` -** change root password -** disable selinux in `/etc/sysconfig/selinux` -** `reboot` -** ask a koji admin to re-enable the host -//// \ No newline at end of file