= Sigul servers upgrades/reboots

Fedora currently has 1 sign-bridge and 2 sign-vault machines for
primary, there is a similar setup for secondary architectures. When
upgrading or rebooting these machines, some special steps must be taken
to ensure everything is working as expected.

== Contact Information

Owner::
  Fedora Release Engineering
Contact::
  #fedora-admin, #fedora-noc
Servers::
  sign-vault03, sign-vault04, sign-bridge02, secondary-bridge01.qa
Purpose::
  Upgrade or restart sign servers

== Description

. Coordinate with releng on timing. Make sure no signing is
happening, and none is planned for a bit.
+
____
Sign-bridge02, secondary-bridge01.qa:
____
+
____
[arabic]
. Apply updates or changes
. Reboot virtual instance
. Once it comes back, start the sigul_bridge service and enter empty
password.
____
+
____
Sign-vault03/04:
____
+
____
[arabic]
. Determine which server is currently primary. It's the one that has
  the floating ip address for sign-vault02 on it.
. Login to the non primary server via serial or management console.
  (There is no ssh access to these servers)
. Take a lvm snapshot:
+
....
lvcreate --size 5G --snapshot --name YYYMMDD /dev/mapper/vg_signvault04-lv_root

Replace YYMMDD with todays year, month, day and the vg with the correct name 
Then apply updates. 
....
.  Confirm the server comes back up ok, login to serial console or management
  console and start the sigul_server process. Enter password when
  prompted.
. On the primary server, down the floating ip address:
+
....
ip addr del 10.5.125.75 dev eth0
....
. On the secondary server, up the floating ip address:
+
....
ip addr add 10.5.125.75 dev eth0
....
. Have rel-eng folks sign some packages to confirm all is working.
. Update/reboot the old primary server and confirm it comes back up ok.
____

[NOTE]
====
Changes to database

When making any changes to the database (new keys, etc), it's important
to sync the data from the primary to the secondary server. This process
is currently manual.
====
