added code as example
Signed-off-by: Pedro Moura <pmoura@redhat.com>
This commit is contained in:
parent
b8a08cdfb5
commit
24b185baab
1 changed files with 11 additions and 132 deletions
|
@ -21,143 +21,22 @@ Follow the process to upgrade builders in staging environment:
|
||||||
|
|
||||||
=== Production Environment
|
=== Production Environment
|
||||||
|
|
||||||
For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From _buildvm-s390x-01_ to _buildvm-s390x-14_ are **zvm** instances, they have to be updated with `dnf --releasever N distro-sync`.
|
For production a similar process is followed, but to avoid outages, builders are disabled, reinstalled and re-added small groups at a time. From _buildvm-s390x-01_ to _buildvm-s390x-14_ are **zvm** instances, they have to be updated with `dnf --releasever NAME distro-sync`.
|
||||||
In production builders, are usually done per bvmhost at a time:
|
In production builders, are usually done per bvmhost at a time:
|
||||||
|
|
||||||
. To do this process the user should be koji admin
|
. To do this process the user should be koji admin
|
||||||
. Find all the builders on the host looking in `/var/log/virthost-lists.out` on batcave01 or look for `vmhost` variable in `inventory/host_vars/` on https://pagure.io/fedora-infra/ansible[ansible repository]
|
. Find all the builders on the host looking in `/var/log/virthost-lists.out` on batcave01 or look for `vmhost` variable in `inventory/host_vars/` on https://pagure.io/fedora-infra/ansible[ansible repository]
|
||||||
. In a loop, disable all those builders using `koji disable-host <vmname>`
|
. In a loop, disable all those builders using `koji disable-host <vmname>`. Like the example below:
|
||||||
|
+
|
||||||
|
....
|
||||||
|
for i in seq -w 01 35
|
||||||
|
do
|
||||||
|
echo $i
|
||||||
|
koji disable-host buildvm-s390x-$i.s390.fedoraproject.org
|
||||||
|
done
|
||||||
|
....
|
||||||
. In a loop check for running builds on them with `koji list-tasks --host <vmname>`
|
. In a loop check for running builds on them with `koji list-tasks --host <vmname>`
|
||||||
. Either wait for all of them to finish or optionally run `koji free-task NNNNN` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers
|
. Either wait for all of them to finish or optionally run `koji free-task NAME` to free a task and another builder will pick it up, but this second option will restart the build and if it is a long running build it could disturb maintainers
|
||||||
. Once they are all empty, destroy them as above and reinstall them with the `buildvm` playbook. Optionally, this is a good time to update the bvmhost and reboot it after destroying, but before installing builders
|
. Once they are all empty, destroy them as above and reinstall them with the `buildvm` playbook. Optionally, this is a good time to update the bvmhost and reboot it after destroying, but before installing builders
|
||||||
. After installed, check that each builder is checking in with `koji list-hosts | grep <vmname>`
|
. After installed, check that each builder is checking in with `koji list-hosts | grep <vmname>`
|
||||||
. In a loop, reenable them
|
. In a loop, reenable them
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
////
|
|
||||||
From this comment below, everything is going to be deleted
|
|
||||||
Keeping all commented for reference while writing a new SOP
|
|
||||||
////
|
|
||||||
|
|
||||||
////
|
|
||||||
= Setup Koji Builder SOP
|
|
||||||
|
|
||||||
== Contents
|
|
||||||
|
|
||||||
* <<_builder_setup>>
|
|
||||||
** <<_network_overview>>
|
|
||||||
** <<_setup_overview>>
|
|
||||||
** <<_resettinginstalling_an_old_koji_builder>>
|
|
||||||
|
|
||||||
== Builder Setup
|
|
||||||
|
|
||||||
Setting up a new koji builder involves a goodly number of steps:
|
|
||||||
|
|
||||||
=== Network Overview
|
|
||||||
|
|
||||||
[arabic]
|
|
||||||
. First get an instance spun up following the kickstart sop.
|
|
||||||
. Define a hostname for it on the 125 network and a $hostname-nfs name
|
|
||||||
for it on the .127 network.
|
|
||||||
. make sure the instance has 2 network connections:
|
|
||||||
* eth0 should be on the .125 network
|
|
||||||
* eth1 should be on the .127 network
|
|
||||||
+
|
|
||||||
[NOTE]
|
|
||||||
=====
|
|
||||||
For VM _eth0_ should be on _br0_, _eth1_ on _br1_ on the vmhost.
|
|
||||||
=====
|
|
||||||
|
|
||||||
=== Setup Overview
|
|
||||||
|
|
||||||
* install the system as normal:
|
|
||||||
+
|
|
||||||
....
|
|
||||||
virt-install -n $builder_fqdn -r $memsize \
|
|
||||||
-f $path_to_lvm --vcpus=$numprocs \
|
|
||||||
-l http://10.5.126.23/repo/rhel/RHEL6-x86_64/ \
|
|
||||||
-x "ksdevice=eth0 ks=http://10.5.126.23/repo/rhel/ks/kvm-rhel-6 \
|
|
||||||
ip=$ip netmask=$netmask gateway=$gw dns=$dns \
|
|
||||||
console=tty0 console=ttyS0" \
|
|
||||||
--network=bridge=br0 --network=bridge=br1 \
|
|
||||||
--vnc --noautoconsole
|
|
||||||
....
|
|
||||||
* run python `/root/tmp/setup-nfs-network.py` this should print out the
|
|
||||||
-nfs hostname that you made above
|
|
||||||
* change root pw
|
|
||||||
* disable selinux on the machine in /etc/sysconfig/selinux
|
|
||||||
* reboot
|
|
||||||
* setup ssl cert into private/builders - use fqdn of host as DN
|
|
||||||
** login to _fas01_ as root
|
|
||||||
** `cd /var/lib/fedora-ca`
|
|
||||||
** `./kojicerthelper.py normal --outdir=/tmp/ \ --name=$fqdn_of_the_new_builder --cadir=. --caname=Fedora`
|
|
||||||
** info for the cert should be like this:
|
|
||||||
+
|
|
||||||
....
|
|
||||||
Country Name (2 letter code) [US]:
|
|
||||||
State or Province Name (full name) [North Carolina]:
|
|
||||||
Locality Name (eg, city) [Raleigh]:
|
|
||||||
Organization Name (eg, company) [Fedora Project]:
|
|
||||||
Organizational Unit Name (eg, section) []:Fedora Builders
|
|
||||||
Common Name (eg, your name or your servers hostname) []:$fqdn_of_new_builder
|
|
||||||
Email Address []:buildsys@fedoraproject.org
|
|
||||||
....
|
|
||||||
** scp the file in `/tmp/$\{fqdn}_key_and_cert.pem` over to _batcave01_
|
|
||||||
** put file in the private repo under `private/builders/$dn}.pem`
|
|
||||||
** `git add` + `git commit`
|
|
||||||
** `git push`
|
|
||||||
* run `./sync-hosts` in infra-hosts repo; `git commit; git push`
|
|
||||||
* as a koji admin run:
|
|
||||||
+
|
|
||||||
....
|
|
||||||
koji add-host $fqdnr i386 x86_64
|
|
||||||
|
|
||||||
(note: those are yum basearchs on the end - season to taste)
|
|
||||||
....
|
|
||||||
|
|
||||||
=== Resetting/installing an old koji builder
|
|
||||||
|
|
||||||
* disable the builder in koji (ask a koji admin)
|
|
||||||
* halt the old system (`halt -p`)
|
|
||||||
* undefine the vm instance on the buildvmhost:
|
|
||||||
+
|
|
||||||
....
|
|
||||||
virsh undefine $builder_fqdn
|
|
||||||
....
|
|
||||||
* reinstall it - from the buildvmhost run:
|
|
||||||
+
|
|
||||||
....
|
|
||||||
virt-install -n $builder_fqdn -r $memsize \
|
|
||||||
-f $path_to_lvm --vcpus=$numprocs \
|
|
||||||
-l http://10.5.126.23/repo/rhel/RHEL6-x86_64/ \
|
|
||||||
-x "ksdevice=eth0 ks=http://10.5.126.23/repo/rhel/ks/kvm-rhel-6 \
|
|
||||||
ip=$ip netmask=$netmask gateway=$gw dns=$dns \
|
|
||||||
console=tty0 console=ttyS0" \
|
|
||||||
--network=bridge=br0 --network=bridge=br1 \
|
|
||||||
--vnc --noautoconsole
|
|
||||||
....
|
|
||||||
* watch install via vnc:
|
|
||||||
+
|
|
||||||
....
|
|
||||||
vncviewer -via bastion.fedoraproject.org $builder_fqdn:1
|
|
||||||
....
|
|
||||||
* when the install finishes:
|
|
||||||
** start the instance on the buildvmhost:
|
|
||||||
+
|
|
||||||
....
|
|
||||||
virsh start $builder_fqdn
|
|
||||||
....
|
|
||||||
** set it to autostart on the buildvmhost:
|
|
||||||
+
|
|
||||||
....
|
|
||||||
virsh autostart $builder_fqdn
|
|
||||||
....
|
|
||||||
* when the guest comes up
|
|
||||||
** login via ssh using the temp root password
|
|
||||||
** python `/root/tmp/setup-nfs-network.py`
|
|
||||||
** change root password
|
|
||||||
** disable selinux in `/etc/sysconfig/selinux`
|
|
||||||
** `reboot`
|
|
||||||
** ask a koji admin to re-enable the host
|
|
||||||
////
|
|
Loading…
Add table
Add a link
Reference in a new issue