= Fedora Status Service - SOP

Fedora-Status is the software that generates the page at
http://status.fedoraproject.org/. This page should be kept up to date
with the current status of the services ran by Fedora Infrastructure.

This page is hosted at AWS.

== Contact Information

Owner:::
  Fedora Infrastructure Team
Contact:::
  #fedora-admin, #fedora-noc
Servers:::
  AWS S3/CloudFront
Purpose:::
  Give status information to users about the current status of our
  public services.
Repository:::
  https://github.com/fedora-infra/statusfpo

== How it works

To keep this website as stable as can be, the page is hosted external to
our main infrastructure, in AWS.

It is based on an S3 bucket with the files, fronted by a CloudFront
distribution for TLS termination and CNAMEs.

The website is statically generated using Pelican on your local machine,
and then pushed to S3.

== Adding and changing outages

=== Making Changes

Before pushing changes live to S3, use the Pelican's devserver to stage
and view changes.

[arabic]
. Install the packages you need to run the devserver with:
+
....
sudo dnf install pelican python-packaging
....
. Check out the repo at:
+
....
git@github.com:fedora-infra/statusfpo.git
....
. Run the devserver with:
+
....
make devserver
....
. View the generated site at http://0.0.0.0:8000. Note that any changes
to the content and theme will automatically regenerate.
. Commit changes (or open a Pull Request) to
https://github.com/fedora-infra/statusfpo

=== Create a new outage

[arabic]
. Add a markdown file to either content/planned/, content/ongoing, or
content/resolved/. The name of the file needs to be unique, so check the
resolved outages for an idea on how to name your file.
. Add your outage notice to the markdown file, for example:
+
....
Title: Buzzilla Slow
Date: 2021-04-28 10:22+0000
OutageFinish: 2021-04-28 13:30+0000
Ticket: 123456

A swarm of bees have taken up residence in one of 
the Buzzilla Server rooms. Consequently, some 
requests to Buzzilla may respond slower than
usual. An apiarist has been called to capture 
and relocate the swarm.
....

____
* Note that OutageFinish is optional, but should really only be ommited
if the projected / or actual outage time is unknown.
* When providing dated, keep the timezone offset at +0000 / UTC
datetimes
____

=== Moving an outage

To move an outage, say from Planned to Ongoing simply move the markdown
file into a different status directory in content/, and regenerate.

== Publishing

Only members of sysadmin-main and people given the AWS credentials can
update the status website.

=== Initial Configuration for Publishing

[arabic]
. First, install the AWS command line tool with:
+
....
sudo dnf install aws-cli
....
. Grab ansible-private/files/aws-status-credentials and store in
~/.aws/credentials.
. Run:
+
....
aws configure set preview.cloudfront true
....

=== Publishing changes live

Once you are satisfied with your changes and how they look on the
devserver, and they have been committed to Git, and push the built
changes live with the command:

....
make upload
....

Note that this command only updates content changes (i.e. adding /
moving outages)

=== Publishing theme changes

If your changes involve changes to the theme, run the following command
to upload everything content and theme changes to the live server:

....
make upload-theme
....

== Renewing SSL certificate

Because

[arabic]
. Run certbot to generate certificate and have it signed by LetsEncrypt
(you can run this command anywhere certbot is installed, you can use
your laptop or certgetter01.phx2.fedoraproject.org):
+
....
rm -rf ~/certbot
certbot certonly --agree-tos -m admin@fedoraproject.org --no-eff-email --manual --manual-public-ip-logging-ok -d status.fedoraproject.org -d www.fedorastatus.org --preferred-challenges http-01 --config-dir ~/certbot/conf --work-dir ~/certbot/work --logs-dir ~/certbot/log
....
. You will be asked to make specific file available under specific URL.
In a different terminal upload requested file to AWS S3 bucket:
+
....
echo SOME_VALUE >myfile
aws --profile statusfpo s3 cp myfile s3://status.fedoraproject.org/.well-known/acme-challenge/SOME_FILE
....
. Verify that uploaded file is available under the rigt URL. If previous
certificate already expired you may need to run curl with -k option:
+
....
curl -kL http://www.fedorastatus.org/.well-known/acme-challenge/SOME_FILE
....
. After making sure that curl outputs expected value, go back to certbot
run and continue by pressing Enter. You will be asked to repeat steps 2
and 3 for another domain. Note that S3 bucket name should stay the same.
. Deploy generated certificate to AWS. This requires additional
permissions on AWS.