2022-06-16 09:47:58 +00:00
|
|
|
= Fedora CoreOS Cincinnati SOP
|
|
|
|
|
|
|
|
|
|
Cincinnati is the update service/backend for Fedora CoreOS (FCOS) machines.
|
|
|
|
|
This SOP describes how to access and how to troubleshoot it.
|
|
|
|
|
|
|
|
|
|
== Contact Information
|
|
|
|
|
|
|
|
|
|
Owner::
|
|
|
|
|
Fedora CoreOS Team
|
|
|
|
|
Contact::
|
|
|
|
|
#fedora-coreos
|
|
|
|
|
|
|
|
|
|
== Details
|
|
|
|
|
|
|
|
|
|
Source::
|
|
|
|
|
https://github.com/coreos/fedora-coreos-cincinnati
|
|
|
|
|
Playbook::
|
|
|
|
|
https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/coreos-cincinnati.yml
|
|
|
|
|
Location::
|
|
|
|
|
OpenShift cluster (production): https://console-openshift-console.apps.ocp.fedoraproject.org/
|
|
|
|
|
Project::
|
|
|
|
|
coreos-cincinnati:
|
|
|
|
|
https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/cluster/projects/coreos-cincinnati
|
|
|
|
|
Deployment::
|
|
|
|
|
https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/coreos-cincinnati/deploymentconfigs/coreos-cincinnati
|
|
|
|
|
Containers::
|
|
|
|
|
* `fcos-graph-builder` (GB - raw updates graph)
|
|
|
|
|
* `fcos-policy-engine` (PE - frontend handling client requests)
|
|
|
|
|
Routes::
|
|
|
|
|
* `coreos-updates-raw` (GB web service)
|
|
|
|
|
* `coreos-updates-raw-status` (GB status and metrics)
|
|
|
|
|
* `coreos-updates` (PE web service)
|
|
|
|
|
* `coreos-updates-status` (PE status and metrics)
|
|
|
|
|
|
|
|
|
|
== Troubleshooting
|
|
|
|
|
|
|
|
|
|
Each FCOS Cincinnati service exposes live metrics in Prometheus format:
|
|
|
|
|
|
|
|
|
|
Graph-builder::
|
|
|
|
|
https://status.raw-updates.coreos.fedoraproject.org/metrics
|
|
|
|
|
Policy-engine::
|
|
|
|
|
https://status.updates.coreos.fedoraproject.org/metrics
|
|
|
|
|
|
|
|
|
|
== Upgrades
|
|
|
|
|
|
|
|
|
|
=== Building a new version
|
|
|
|
|
|
|
|
|
|
FCOS Cincinnati is built as container image directly from source,
|
|
|
|
|
referencing a pinned git commit.
|
|
|
|
|
|
|
|
|
|
In order to build a new version, you will first have to find the
|
|
|
|
|
relevant commit (i.e. the latest on the `main` branch) at
|
|
|
|
|
https://github.com/coreos/fedora-coreos-cincinnati .
|
|
|
|
|
|
|
|
|
|
Once you have identified the target commit, these are the steps to build
|
|
|
|
|
a new container image:
|
|
|
|
|
|
|
|
|
|
* update the `fcos_cincinnati_build_git_sha` playbook variable in
|
|
|
|
|
`roles/openshift-apps/coreos-cincinnati/vars/staging.yml`
|
|
|
|
|
* update the `fcos_cincinnati_build_git_sha` playbook variable in
|
|
|
|
|
`roles/openshift-apps/coreos-cincinnati/vars/production.yml`
|
|
|
|
|
* commit and push the update to the `fedora-infra/ansible` repository
|
|
|
|
|
* SSH to `batcave01.iad2.fedoraproject.org`
|
|
|
|
|
* run `sudo rbac-playbook openshift-apps/coreos-cincinnati.yml` using
|
|
|
|
|
your FAS password and your second-factor OTP
|
|
|
|
|
* schedule a new build by running
|
|
|
|
|
`sudo rbac-playbook -t build openshift-apps/coreos-cincinnati.yml`
|
|
|
|
|
|
2022-06-27 16:05:37 -04:00
|
|
|
=== Deploying a new version
|
|
|
|
|
|
|
|
|
|
Once the target commit has been built into a container image, these are
|
|
|
|
|
the steps to deploy the new image:
|
|
|
|
|
|
|
|
|
|
* update the `fcos_cincinnati_deploy_git_sha` playbook variable in
|
|
|
|
|
`roles/openshift-apps/coreos-cincinnati/vars/staging.yml`
|
|
|
|
|
* update the `fcos_cincinnati_deploy_git_sha` playbook variable in
|
|
|
|
|
`roles/openshift-apps/coreos-cincinnati/vars/production.yml`
|
|
|
|
|
* commit and push the update to the `fedora-infra/ansible` repository
|
|
|
|
|
* SSH to `batcave01.iad2.fedoraproject.org`
|
|
|
|
|
* run `sudo rbac-playbook openshift-apps/coreos-cincinnati.yml` using
|
|
|
|
|
your FAS password and your second-factor OTP
|
|
|
|
|
|
2022-06-16 09:47:58 +00:00
|
|
|
== Things that could go wrong
|
|
|
|
|
|
|
|
|
|
=== Application build is stuck
|
|
|
|
|
|
|
|
|
|
Issues in the underlying OpenShift cluster may result in builds being
|
|
|
|
|
permanently stuck.
|
|
|
|
|
|
|
|
|
|
If a build does not complete within a reasonable amount of time (i.e. 15
|
|
|
|
|
minutes):
|
|
|
|
|
|
|
|
|
|
* go to the build overview at https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/coreos-cincinnati/builds
|
|
|
|
|
* click on the build
|
|
|
|
|
* cancel it through the "Cancel Build" button
|
|
|
|
|
* go back to the build overview page
|
|
|
|
|
* schedule a new build through the "Start Build" button
|
