sysadmin_guide: add coreos-cincinnati page

This adds a basic SOP page for the Fedora CoreOS Cincinnati service
running on Fedora OCP4 cluster.

modules/sysadmin_guide/pages/coreos-cincinnati.adoc

@@ -0,0 +1,83 @@
+= Fedora CoreOS Cincinnati SOP
+
+Cincinnati is the update service/backend for Fedora CoreOS (FCOS) machines.
+This SOP describes how to access and how to troubleshoot it.
+
+== Contact Information
+
+Owner::
+  Fedora CoreOS Team
+Contact::
+  #fedora-coreos
+
+== Details
+
+Source::
+  https://github.com/coreos/fedora-coreos-cincinnati
+Playbook::
+  https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/coreos-cincinnati.yml
+Location::
+  OpenShift cluster (production): https://console-openshift-console.apps.ocp.fedoraproject.org/
+Project::
+  coreos-cincinnati:
+  https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/cluster/projects/coreos-cincinnati
+Deployment::
+  https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/coreos-cincinnati/deploymentconfigs/coreos-cincinnati
+Containers::
+  * `fcos-graph-builder` (GB - raw updates graph)
+  * `fcos-policy-engine` (PE - frontend handling client requests)
+Routes::
+  * `coreos-updates-raw` (GB web service)
+  * `coreos-updates-raw-status` (GB status and metrics)
+  * `coreos-updates` (PE web service)
+  * `coreos-updates-status` (PE status and metrics)
+
+== Troubleshooting
+
+Each FCOS Cincinnati service exposes live metrics in Prometheus format:
+
+Graph-builder::
+  https://status.raw-updates.coreos.fedoraproject.org/metrics
+Policy-engine::
+  https://status.updates.coreos.fedoraproject.org/metrics
+
+== Upgrades
+
+=== Building a new version
+
+FCOS Cincinnati is built as container image directly from source,
+referencing a pinned git commit.
+
+In order to build a new version, you will first have to find the
+relevant commit (i.e. the latest on the `main` branch) at
+https://github.com/coreos/fedora-coreos-cincinnati .
+
+Once you have identified the target commit, these are the steps to build
+a new container image:
+
+  * update the `fcos_cincinnati_build_git_sha` playbook variable in
+    `roles/openshift-apps/coreos-cincinnati/vars/staging.yml`
+  * update the `fcos_cincinnati_build_git_sha` playbook variable in
+    `roles/openshift-apps/coreos-cincinnati/vars/production.yml`
+  * commit and push the update to the `fedora-infra/ansible` repository
+  * SSH to `batcave01.iad2.fedoraproject.org`
+  * run `sudo rbac-playbook openshift-apps/coreos-cincinnati.yml` using
+    your FAS password and your second-factor OTP
+  * schedule a new build by running
+    `sudo rbac-playbook -t build openshift-apps/coreos-cincinnati.yml`
+
+== Things that could go wrong
+
+=== Application build is stuck
+
+Issues in the underlying OpenShift cluster may result in builds being
+permanently stuck.
+
+If a build does not complete within a reasonable amount of time (i.e. 15
+minutes):
+
+  * go to the build overview at https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/coreos-cincinnati/builds
+  * click on the build
+  * cancel it through the "Cancel Build" button
+  * go back to the build overview page
+  * schedule a new build through the "Start Build" button