From 43acdf5d71e651d500c6cf11c8a7460155c2e6c9 Mon Sep 17 00:00:00 2001
From: Luca BRUNO <luca.bruno@coreos.com>
Date: Thu, 16 Jun 2022 09:47:58 +0000
Subject: [PATCH] sysadmin_guide: add coreos-cincinnati page

This adds a basic SOP page for the Fedora CoreOS Cincinnati service
running on Fedora OCP4 cluster.
---
 modules/sysadmin_guide/nav.adoc               |  1 +
 .../pages/coreos-cincinnati.adoc              | 83 +++++++++++++++++++
 2 files changed, 84 insertions(+)
 create mode 100644 modules/sysadmin_guide/pages/coreos-cincinnati.adoc

diff --git a/modules/sysadmin_guide/nav.adoc b/modules/sysadmin_guide/nav.adoc
index 9408d99..a72d8e5 100644
--- a/modules/sysadmin_guide/nav.adoc
+++ b/modules/sysadmin_guide/nav.adoc
@@ -18,6 +18,7 @@
 ** xref:compose-tracker.adoc[Compose Tracker - SOP]
 ** xref:contenthosting.adoc[Content Hosting Infrastructure - SOP]
 ** xref:copr.adoc[Copr - SOP]
+** xref:coreos-cincinnati.adoc[CoreOS Cincinnati - SOP]
 ** xref:database.adoc[Database Infrastructure - SOP]
 ** xref:datanommer.adoc[datanommer - SOP]
 ** xref:debuginfod.adoc[Fedora Debuginfod Service - SOP]
diff --git a/modules/sysadmin_guide/pages/coreos-cincinnati.adoc b/modules/sysadmin_guide/pages/coreos-cincinnati.adoc
new file mode 100644
index 0000000..ca3289e
--- /dev/null
+++ b/modules/sysadmin_guide/pages/coreos-cincinnati.adoc
@@ -0,0 +1,83 @@
+= Fedora CoreOS Cincinnati SOP
+
+Cincinnati is the update service/backend for Fedora CoreOS (FCOS) machines.
+This SOP describes how to access and how to troubleshoot it.
+
+== Contact Information
+
+Owner::
+  Fedora CoreOS Team
+Contact::
+  #fedora-coreos
+
+== Details
+
+Source::
+  https://github.com/coreos/fedora-coreos-cincinnati
+Playbook::
+  https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/coreos-cincinnati.yml
+Location::
+  OpenShift cluster (production): https://console-openshift-console.apps.ocp.fedoraproject.org/
+Project::
+  coreos-cincinnati:
+  https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/cluster/projects/coreos-cincinnati
+Deployment::
+  https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/coreos-cincinnati/deploymentconfigs/coreos-cincinnati
+Containers::
+  * `fcos-graph-builder` (GB - raw updates graph)
+  * `fcos-policy-engine` (PE - frontend handling client requests)
+Routes::
+  * `coreos-updates-raw` (GB web service)
+  * `coreos-updates-raw-status` (GB status and metrics)
+  * `coreos-updates` (PE web service)
+  * `coreos-updates-status` (PE status and metrics)
+
+== Troubleshooting
+
+Each FCOS Cincinnati service exposes live metrics in Prometheus format:
+
+Graph-builder::
+  https://status.raw-updates.coreos.fedoraproject.org/metrics
+Policy-engine::
+  https://status.updates.coreos.fedoraproject.org/metrics
+
+== Upgrades
+
+=== Building a new version
+
+FCOS Cincinnati is built as container image directly from source,
+referencing a pinned git commit.
+
+In order to build a new version, you will first have to find the
+relevant commit (i.e. the latest on the `main` branch) at
+https://github.com/coreos/fedora-coreos-cincinnati .
+
+Once you have identified the target commit, these are the steps to build
+a new container image:
+
+  * update the `fcos_cincinnati_build_git_sha` playbook variable in
+    `roles/openshift-apps/coreos-cincinnati/vars/staging.yml`
+  * update the `fcos_cincinnati_build_git_sha` playbook variable in
+    `roles/openshift-apps/coreos-cincinnati/vars/production.yml`
+  * commit and push the update to the `fedora-infra/ansible` repository
+  * SSH to `batcave01.iad2.fedoraproject.org`
+  * run `sudo rbac-playbook openshift-apps/coreos-cincinnati.yml` using
+    your FAS password and your second-factor OTP
+  * schedule a new build by running
+    `sudo rbac-playbook -t build openshift-apps/coreos-cincinnati.yml`
+
+== Things that could go wrong
+
+=== Application build is stuck
+
+Issues in the underlying OpenShift cluster may result in builds being
+permanently stuck.
+
+If a build does not complete within a reasonable amount of time (i.e. 15
+minutes):
+
+  * go to the build overview at https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/coreos-cincinnati/builds
+  * click on the build
+  * cancel it through the "Cancel Build" button
+  * go back to the build overview page
+  * schedule a new build through the "Start Build" button