Added the infra SOPs ported to asciidoc.

modules/sysadmin_guide/pages/sigul-upgrade.adoc

@@ -0,0 +1,82 @@
+= Sigul servers upgrades/reboots
+
+Fedora currently has 1 sign-bridge and 2 sign-vault machines for
+primary, there is a similar setup for secondary architectures. When
+upgrading or rebooting these machines, some special steps must be taken
+to ensure everything is working as expected.
+
+== Contact Information
+
+Owner::
+  Fedora Release Engineering
+Contact::
+  #fedora-admin, #fedora-noc
+Servers::
+  sign-vault03, sign-vault04, sign-bridge02, secondary-bridge01.qa
+Purpose::
+  Upgrade or restart sign servers
+
+== Description
+
+{empty}0. Coordinate with releng on timing. Make sure no signing is
+happening, and none is planned for a bit.
+
+Sign-bridge02, secondary-bridge01.qa:
+
+____
+[arabic]
+. Apply updates or changes
+. Reboot virtual instance
+. Once it comes back, start the sigul_bridge service and enter empty
+password.
+____
+
+Sign-vault03/04:
+
+____
+[arabic]
+. {blank}
++
+Determine which server is currently primary. It's the one that has the::
+  floating ip address for sign-vault02 on it.
+. {blank}
++
+Login to the non primary server via serial or management console.::
+  (There is no ssh access to these servers)
+. Take a lvm snapshot:
++
+....
+lvcreate --size 5G --snapshot --name YYYMMDD /dev/mapper/vg_signvault04-lv_root
+
+Replace YYMMDD with todays year, month, day and the vg with the correct name 
+Then apply updates. 
+....
+. {blank}
++
+Confirm the server comes back up ok, login to serial console or
+management::
+  console and start the sigul_server process. Enter password when
+  prompted.
+. On the primary server, down the floating ip address:
++
+....
+ip addr del 10.5.125.75 dev eth0
+....
+. On the secondary server, up the floating ip address:
++
+....
+ip addr add 10.5.125.75 dev eth0
+....
+. Have rel-eng folks sign some packages to confirm all is working.
+. Update/reboot the old primary server and confirm it comes back up ok.
+____
+
+[NOTE]
+.Note
+====
+Changes to database
+
+When making any changes to the database (new keys, etc), it's important
+to sync the data from the primary to the secondary server. This process
+is currently manual.
+====