= OpenShift SOP

OpenShift is used in Fedora Infrastructure to host a number of
applications. This SOP is applicable to the OpenShift cluster and not
the application running on it.

Production instance: https://os.fedoraproject.org/

Staging instance: https://os.stg.fedoraproject.org/

== Contact information

Owner::
  Fedora Infrastrucutre Team
Contact::
  #fedora-admin
Persons::
  .oncall
Servers::
  * os-master01.iad2.fedoraproject.org
  * os-master02.iad2.fedoraproject.org
  * os-master03.iad2.fedoraproject.org
  * os-node01.iad2.fedoraproject.org
  * os-node02.iad2.fedoraproject.org
  * os-node03.iad2.fedoraproject.org
  * os-node04.iad2.fedoraproject.org
  * os-node05.iad2.fedoraproject.org
Purpose::
  Run Fedora Infrastructure applications

== Things that could go wrong

=== Application build is stuck

If an application build seems stuck, it generally helps to restart the
docker service on the node used for the build.

First check which builds are currently running on the cluster:

....
[os-master01] # oc get builds --all-namespaces | grep -i running
....

If the builds seems stuck (ie running for more than 20 mins) check on
which nodes they are scheduled.

Let's take a bodhi build for example:

....
[os-master01] # oc -n bodhi get builds

[os-master01] # oc -n bodhi describe build bodhi-base-49 | grep os-node
....

Once you have identified which node the build is running on, you can
restart the docker service on this node:

....
[os-node02] # systemctl restart docker
....

You can start a new build:

....
[os-master01] # oc -n bodhi start-build bodhi-base
....

Finally you can check if there are any more build stuck. If that's the
case just repeat these steps:

....
[os-master01] # oc get builds --all-namespaces
....