= FedMsg Notifications (FMN) SOP

Route individualized notifications to fedora contributors over email,
irc.

== Contact Information

=== Owner

* Messaging SIG
* Fedora Infrastructure Team

=== Contact

____
* #fedora-apps for FMN development
* #fedora-fedmsg for an IRC feed of all fedmsgs
* #fedora-admin for problems with the deployment of FMN
* #fedora-noc for outage/crisis alerts
____

=== Servers

Production servers:

____
* notifs-backend01.iad2.fedoraproject.org (RHEL 7)
* notifs-web01.iad2.fedoraproject.org (RHEL 7)
* notifs-web02.iad2.fedoraproject.org (RHEL 7)
____

Staging servers:

____
* notifs-backend01.stg.iad2.fedoraproject.org (RHEL 7)
* notifs-web01.stg.iad2.fedoraproject.org (RHEL 7)
* notifs-web02.stg.iad2.fedoraproject.org (RHEL 7)
____

=== Purpose

Route notifications to users

== Description

fmn is a pair of systems intended to route fedmsg notifications to
Fedora contributors and users.

There is a web interface running on notifs-web01 and notifs-web02 that
allows users to login and configure their preferences to select this or
that type of message.

There is a backend running on notifs-backend01 where most of the work is
done.

The backend process is a 'fedmsg-hub' daemon, controlled by systemd.

== Hosts

=== notifs-backend

This host runs:

* `fedmsg-hub.service`
* One or more `fmn-worker@.service`. Currently notifs-backend01 runs
`fmn-worker@\{1-4}.service`
* `fmn-backend@1.service`
* `fmn-digests@1.service`
* `rabbitmq-server.service`, an AMQP broker used to communicate between
the services.
* `redis.service`, used for caching.

This host relies on a PostgreSQL database running on
db01.phx2.fedoraproject.org.

=== notifs-web

This host runs:

* A Python WSGI application via Apache httpd that serves the
https://apps.fedoraproject.org/notifications[FMN web user interface].

This host relies on a PostgreSQL database running on
db01.iad2.fedoraproject.org.

== Deployment

Once upstream releases a new version of
https://github.com/fedora-infra/fmn[fmn],
https://github.com/fedora-infra/fmn.web[fmn-web], or
https://github.com/fedora-infra/fmn.sse[fmn-sse] creating a Git tag, a
new version can be built an deployed into Fedora infrastructure.

=== Building

FMN is packaged in Fedora and EPEL as
https://src.fedoraproject.org/rpms/python-fmn/[python-fmn]
(the backend),
https://src.fedoraproject.org/rpms/python-fmn-web/[python-fmn-web]
(the frontend), and the optional
https://src.fedoraproject.org/rpms/python-fmn-sse/[python-fmn-sse].

Since all the hosts run RHEL 7, you need to build all these packages for
EPEL 7.

=== Configuration

If there are any configuration updates required by the new version of
FMN, update the `notifs` Ansible roles on
batcave01.iad2.fedoraproject.org. Remember to use:

....
{% if env == 'staging' %}
    <new config here>
{% else %}
    <retain old config>
{% endif %}
....

When deploying the update to staging. You can apply configuration
updates to staging by running:

....
$ sudo rbac-playbook -l staging groups/notifs-backend.yml
$ sudo rbac-playbook -l staging groups/notifs-web.yml
....

Simply drop the `-l staging` to update the production configuration.

=== Upgrading

To upgrade the
https://src.fedoraproject.org/rpms/python-fmn/[python-fmn],
https://src.fedoraproject.org/rpms/python-fmn-web/[python-fmn-web],
and
https://src.fedoraproject.org/rpms/python-fmn-sse/[python-fmn-sse]
packages, apply configuration changes, and restart the services, you
should use the manual upgrade playbook:

....
$ sudo rbac-playbook -l staging manual/upgrade/fmn.yml
....

Again, drop the `-l staging` flag to upgrade production.

Be aware that the FMN services take a significant amount of time to
start up as they pre-heat their caches before starting work.

== Service Administration

Disable an account (on notifs-backend01):

....
$ sudo -u fedmsg /usr/local/bin/fmn-disable-account USERNAME
....

Restart:

....
$ sudo systemctl restart fedmsg-hub
....

Watch logs:

....
$ sudo journalctl -u fedmsg-hub -f
....

Configuration:

....
$ ls /etc/fedmsg.d/
$ sudo fedmsg-config | less
....

Upgrade (from batcave):

....
$ sudo -i ansible-playbook /srv/web/infra/ansible/playbooks/manual/upgrade/fmn.yml
....

== Mailing Lists

We use FMN as a way to forward certain kinds of messages to mailing
lists so people can read them the good old fashioned way that they like
to. To accomplish this, we create 'bot' FAS accounts with their own FMN
profiles and we set their email addresses to the lists in question.

If you need to change the way some set of messages are forwarded, you
can do it from the FMN web interface (if you are an FMN admin as defined
in the config file in roles/notifs/frontend/). You can navigate to
https://apps.fedoraproject.org/notifications/USERNAME.id.fedoraproject.org
to do this.

If the account exists as a FAS user already (for instance, the
`virtmaint` user) but it does not yet exist in FMN, you can add it to
the FMN database by logging in to notifs-backend01 and running
`fmn-create-user --email DESTINATION@EMAIL.COM --create-defaults FAS_USERNAME`.