Infrastructure/infra-docs-fpo
1
0
Fork 0
Code Issues 100 Pull requests 6 Projects Releases Packages Wiki Activity Actions

Review massupgrade SOP

Browse source 
Signed-off-by: Michal Konečný <mkonecny@redhat.com>
This commit is contained in:
Michal Konečný 2021-09-06 16:44:44 +02:00
parent 7675a5e0b4
commit 13e9b3d32c
2 changed files with 106 additions and 112 deletions
Download patch file Download diff file Expand all files Collapse all files

2
modules/sysadmin_guide/nav.adoc
View file

@ -65,7 +65,7 @@
** xref:layered-image-buildsys.adoc[Layered Image Build System - SOP]
** xref:mailman.adoc[Mailman Infrastructure - SOP]
** xref:making-ssl-certificates.adoc[SSL Certificate Creation - SOP]
** xref:massupgrade.adoc[massupgrade - SOP in review ]
** xref:massupgrade.adoc[Mass Upgrade Infrastructure - SOP]
** xref:mastermirror.adoc[mastermirror - SOP in review ]
** xref:mbs.adoc[mbs - SOP in review ]
** xref:memcached.adoc[memcached - SOP in review ]

216
modules/sysadmin_guide/pages/massupgrade.adoc
View file

@ -5,26 +5,22 @@ various security and other upgrades.
== Contents
[arabic]
. Contact Information
. Preparation
. Staging
. Special Considerations
+
____
* Disable builders
* Post reboot action
* Schedule autoqa01 reboot
* Bastion01 and Bastion02 and openvpn server
* Special yum directives
____
. Update Leader
. Group A reboots
. Group B reboots
. Group C reboots
. Doing the upgrade
. Doing the reboot
. Aftermath
* <<_contact_information>>
* <<_preparation>>
* <<_staging>>
* <<_special_considerations>>
** <<_disable_builders>>
** <<_post_reboot_action>>
** <<_schedule_autoqa01_reboot>>
** <<_bastion01_and_bastion02_and_openvpn_server>>
** <<_special_yum_directives>>
* <<_update_leader>>
* <<_group_a_reboots>>
* <<_group_b_reboots>>
* <<_group_c_reboots>>
* <<_doing_the_upgrade>>
* <<_doing_the_reboot>>
* <<_aftermath>>
== Contact Information
@ -56,7 +52,7 @@ Group "C"::
servers that infrastructure will notice are down, or are redundent
enough to reboot some with others taking the load.
. Appoint an 'Update Leader' for the updates.
. Follow the [61]Outage Infrastructure SOP and send advance notification
. Follow the xref:outage.adoc[Outage Infrastructure SOP] and send advance notification
to the appropriate lists. Try to schedule the update at a time when many
admins are around to help/watch for problems and when impact for the
group affected is less. Do NOT do multiple groups on the same day if
@ -112,10 +108,10 @@ koji enable-host NAME
....
[NOTE]
.Note
====
you must be a koji admin
====
Additionally, rel-eng and builder boxes may need a special version
of rpm. Make sure to check with rel-eng on any rpm upgrades for them.
@ -142,12 +138,12 @@ iptables -t nat -I POSTROUTING -s 192.168.122.3/32 -j SNAT --to-source 66.135.62
....
[NOTE]
.Note
====
The source is the internal guest ips, the to-source is the external ips
that map to that guest ip. If there are multiple guests, each one needs
the above SNAT rule inserted.
====
=== Schedule autoqa01 reboot
There is currently an autoqa01.c host on cnode01. Check with QA folks
@ -157,7 +153,7 @@ before rebooting this guest/host.
We need one of the bastion machines to be up to provide openvpn for all
machines. Before rebooting bastion02, modify:
`manifests/nodes/bastion0*.phx2.fedoraproject.org.pp` files to start
`manifests/nodes/bastion0*.iad2.fedoraproject.org.pp` files to start
openvpn server on bastion01, wait for all clients to re-connect, reboot
bastion02 and then revert back to it as openvpn hub.
@ -166,7 +162,8 @@ bastion02 and then revert back to it as openvpn hub.
Sometimes we will wish to exclude or otherwise modify the yum.conf on a
machine. For this purpose, all machines have an include, making them
read
[62]http://infrastructure.fedoraproject.org/infra/hosts/FQHN/yum.conf.include
http://infrastructure.fedoraproject.org/infra/hosts/FQHN/yum.conf.include
(TODO Fix link)
from the infrastructure repo. If you need to make such changes, add them
to the infrastructure repo before doing updates.
@ -197,35 +194,35 @@ These hosts are grouped based on the virt host they reside on:
* ibiblio03.fedoraproject.org
* collab01.fedoraproject.org
* serverbeach09.fedoraproject.org
* db05.phx2.fedoraproject.org
* virthost03.phx2.fedoraproject.org
* db01.phx2.fedoraproject.org
* virthost04.phx2.fedoraproject.org
* db-fas01.phx2.fedoraproject.org
* proxy01.phx2.fedoraproject.org
* virthost05.phx2.fedoraproject.org
* ask01.phx2.fedoraproject.org
* virthost06.phx2.fedoraproject.org
* db05.iad2.fedoraproject.org
* virthost03.iad2.fedoraproject.org
* db01.iad2.fedoraproject.org
* virthost04.iad2.fedoraproject.org
* db-fas01.iad2.fedoraproject.org
* proxy01.iad2.fedoraproject.org
* virthost05.iad2.fedoraproject.org
* ask01.iad2.fedoraproject.org
* virthost06.iad2.fedoraproject.org
These are the rest:
* bapp02.phx2.fedoraproject.org
* bastion02.phx2.fedoraproject.org
* bapp02.iad2.fedoraproject.org
* bastion02.iad2.fedoraproject.org
* app05.fedoraproject.org
* backup02.fedoraproject.org
* bastion01.phx2.fedoraproject.org
* fas01.phx2.fedoraproject.org
* fas02.phx2.fedoraproject.org
* log02.phx2.fedoraproject.org
* memcached03.phx2.fedoraproject.org
* noc01.phx2.fedoraproject.org
* bastion01.iad2.fedoraproject.org
* fas01.iad2.fedoraproject.org
* fas02.iad2.fedoraproject.org
* log02.iad2.fedoraproject.org
* memcached03.iad2.fedoraproject.org
* noc01.iad2.fedoraproject.org
* ns02.fedoraproject.org
* ns04.phx2.fedoraproject.org
* ns04.iad2.fedoraproject.org
* proxy04.fedoraproject.org
* smtp-mm03.fedoraproject.org
* batcave02.phx2.fedoraproject.org
* batcave02.iad2.fedoraproject.org
* mm3test.fedoraproject.org
* packages02.phx2.fedoraproject.org
* packages02.iad2.fedoraproject.org
=== Group B reboots
@ -235,20 +232,20 @@ devel-announce list.
These hosts are grouped based on the virt host they reside on:
* db04.phx2.fedoraproject.org
* bvirthost01.phx2.fedoraproject.org
* nfs01.phx2.fedoraproject.org
* bvirthost02.phx2.fedoraproject.org
* pkgs01.phx2.fedoraproject.org
* bvirthost03.phx2.fedoraproject.org
* kojipkgs02.phx2.fedoraproject.org
* bvirthost04.phx2.fedoraproject.org
* db04.iad2.fedoraproject.org
* bvirthost01.iad2.fedoraproject.org
* nfs01.iad2.fedoraproject.org
* bvirthost02.iad2.fedoraproject.org
* pkgs01.iad2.fedoraproject.org
* bvirthost03.iad2.fedoraproject.org
* kojipkgs02.iad2.fedoraproject.org
* bvirthost04.iad2.fedoraproject.org
These are the rest:
* koji04.phx2.fedoraproject.org
* releng03.phx2.fedoraproject.org
* releng04.phx2.fedoraproject.org
* koji04.iad2.fedoraproject.org
* releng03.iad2.fedoraproject.org
* releng04.iad2.fedoraproject.org
=== Group C reboots
@ -280,69 +277,68 @@ Group C hosts that have proxy servers on them:
* app08.fedoraproject.org
* proxy08.fedoraproject.org
* coloamer01.fedoraproject.org
+
____
Other Group C hosts:
____
* ask01.stg.phx2.fedoraproject.org
* app02.stg.phx2.fedoraproject.org
* proxy01.stg.phx2.fedoraproject.org
* releng01.stg.phx2.fedoraproject.org
* value01.stg.phx2.fedoraproject.org
* virthost13.phx2.fedoraproject.org
* db-fas01.stg.phx2.fedoraproject.org
* pkgs01.stg.phx2.fedoraproject.org
* packages01.stg.phx2.fedoraproject.org
* virthost11.phx2.fedoraproject.org
* app01.stg.phx2.fedoraproject.org
* koji01.stg.phx2.fedoraproject.org
* db02.stg.phx2.fedoraproject.org
* fas01.stg.phx2.fedoraproject.org
* virthost10.phx2.fedoraproject.org
* ask01.stg.iad2.fedoraproject.org
* app02.stg.iad2.fedoraproject.org
* proxy01.stg.iad2.fedoraproject.org
* releng01.stg.iad2.fedoraproject.org
* value01.stg.iad2.fedoraproject.org
* virthost13.iad2.fedoraproject.org
* db-fas01.stg.iad2.fedoraproject.org
* pkgs01.stg.iad2.fedoraproject.org
* packages01.stg.iad2.fedoraproject.org
* virthost11.iad2.fedoraproject.org
* app01.stg.iad2.fedoraproject.org
* koji01.stg.iad2.fedoraproject.org
* db02.stg.iad2.fedoraproject.org
* fas01.stg.iad2.fedoraproject.org
* virthost10.iad2.fedoraproject.org
* autoqa01.qa.fedoraproject.org
* autoqa-stg01.qa.fedoraproject.org
* bastion-comm01.qa.fedoraproject.org
* batcave-comm01.qa.fedoraproject.org
* virthost-comm01.qa.fedoraproject.org
* compose-x86-01.phx2.fedoraproject.org
* compose-x86-02.phx2.fedoraproject.org
* download01.phx2.fedoraproject.org
* download02.phx2.fedoraproject.org
* download03.phx2.fedoraproject.org
* download04.phx2.fedoraproject.org
* download05.phx2.fedoraproject.org
* compose-x86-01.iad2.fedoraproject.org
* compose-x86-02.iad2.fedoraproject.org
* download01.iad2.fedoraproject.org
* download02.iad2.fedoraproject.org
* download03.iad2.fedoraproject.org
* download04.iad2.fedoraproject.org
* download05.iad2.fedoraproject.org
* download-rdu01.vpn.fedoraproject.org
* download-rdu02.vpn.fedoraproject.org
* download-rdu03.vpn.fedoraproject.org
* fas03.phx2.fedoraproject.org
* secondary01.phx2.fedoraproject.org
* memcached04.phx2.fedoraproject.org
* virthost01.phx2.fedoraproject.org
* app02.phx2.fedoraproject.org
* value03.phx2.fedoraproject.org
* virthost07.phx2.fedoraproject.org
* app03.phx2.fedoraproject.org
* value04.phx2.fedoraproject.org
* ns03.phx2.fedoraproject.org
* darkserver01.phx2.fedoraproject.org
* virthost08.phx2.fedoraproject.org
* app04.phx2.fedoraproject.org
* packages02.phx2.fedoraproject.org
* virthost09.phx2.fedoraproject.org
* fas03.iad2.fedoraproject.org
* secondary01.iad2.fedoraproject.org
* memcached04.iad2.fedoraproject.org
* virthost01.iad2.fedoraproject.org
* app02.iad2.fedoraproject.org
* value03.iad2.fedoraproject.org
* virthost07.iad2.fedoraproject.org
* app03.iad2.fedoraproject.org
* value04.iad2.fedoraproject.org
* ns03.iad2.fedoraproject.org
* darkserver01.iad2.fedoraproject.org
* virthost08.iad2.fedoraproject.org
* app04.iad2.fedoraproject.org
* packages02.iad2.fedoraproject.org
* virthost09.iad2.fedoraproject.org
* hosted03.fedoraproject.org
* serverbeach06.fedoraproject.org
* hosted04.fedoraproject.org
* serverbeach07.fedoraproject.org
* collab02.fedoraproject.org
* serverbeach08.fedoraproject.org
* dhcp01.phx2.fedoraproject.org
* relepel01.phx2.fedoraproject.org
* sign-bridge02.phx2.fedoraproject.org
* koji03.phx2.fedoraproject.org
* bvirthost05.phx2.fedoraproject.org
* dhcp01.iad2.fedoraproject.org
* relepel01.iad2.fedoraproject.org
* sign-bridge02.iad2.fedoraproject.org
* koji03.iad2.fedoraproject.org
* bvirthost05.iad2.fedoraproject.org
* (disable each builder in turn, update and reenable).
* ppc11.phx2.fedoraproject.org
* ppc12.phx2.fedoraproject.org
* ppc11.iad2.fedoraproject.org
* ppc12.iad2.fedoraproject.org
* backup03
== Doing the upgrade
@ -350,7 +346,7 @@ ____
If possible, system upgrades should be done in advance of the reboot
(with relevant testing of new packages on staging). To do the upgrades,
make sure that the Infrastructure RHEL repo is updated as necessary to
pull in the new packages ([63]Infrastructure Yum Repo SOP)
pull in the new packages (xref:infra-repo.adoc[Infrastructure Yum Repo SOP])
On batcave01, as root run:
@ -379,7 +375,7 @@ sudo func-command --timeout=10 --oneline /usr/local/bin/needs-reboot.py after-up
In the order determined above, reboots will usually be grouped by the
virtualization hosts that the servers are on. You can see the guests per
virt host on batcave01 in /var/log/virthost-lists.out
virt host on batcave01 in `/var/log/virthost-lists.out`
To reboot sets of boxes based on which virthost they are we've written a
special script which facilitates it:
@ -391,7 +387,7 @@ func-vhost-reboot virthost-fqdn
ex:
....
sudo func-vhost-reboot virthost13.phx2.fedoraproject.org
sudo func-vhost-reboot virthost13.iad2.fedoraproject.org
....
== Aftermath
@ -399,13 +395,11 @@ sudo func-vhost-reboot virthost13.phx2.fedoraproject.org
[arabic]
. Make sure that everything's running fine
. Reenable nagios notification as needed
. {blank}
+
Make sure to perform any manual post-boot setup (such as entering::
. Make sure to perform any manual post-boot setup (such as entering
passphrases for encrypted volumes)
. Close outage ticket.
=== Non virthost reboots:
=== Non virthost reboots
If you need to reboot specific hosts and make sure they recover -
consider using:
@ -415,4 +409,4 @@ sudo func-host-reboot hostname hostname1 hostname2 ...
....
If you want to reboot the hosts one at a time waiting for each to come
back before rebooting the next pass a -o to func-host-reboot.
back before rebooting the next pass a `-o` to `func-host-reboot`.