arc/docs/fmn/april2022/index.rst

FMN Replacement (April 2022)
============================

FMN is a web application allowing users to create filters on messages sent to fedmsg and
forward these as notifications on to email or IRC

- `current deployment <https://apps.fedoraproject.org/notifications/>`__
- `FMN source <https://github.com/fedora-infra/fmn>`__
- `CPE initiative ticket <https://pagure.io/cpe/initiatives-proposal/issue/10>`__

Issues to solve
---------------

Message Delivery Lag
~~~~~~~~~~~~~~~~~~~~

In times of peak congestion on the message queue (e.g. during a mass rebuild) it can
take **days** for messages to be delivered to users.

fedmsg
~~~~~~

The current version of FMN was implemented uses the deprecated fedmsg bus, not the
currently supported Fedora Messaging (rabbitmq) message queue.

Additionally, the FMN code is deeply integrated with the fedmsg ecosystem (e.g. the
configuration setup for FMN is kept in the fedmsg.d/ directory)

UI
--

After adhoc discussions with the development community one of the major issues is the
FMN web UI:

- The interface for creating new filters is complicated and confusing.
- Users have to define filters seperately for different destentations (i.e. email and
  IRC) This is due to how the database is designed.

Design options
--------------

- jcline’s option: write a RabbitMQ extension. While it would probably be way more
  efficient we don’t have the knowledge to write or maintain this code. Also, it would
  require adding headers for the fields we want to filter on, such as usernames and
  packages, and we’ll probably loose in flexibility.
- write a fedora-messaging consumer that would triage incoming message and produce add
  notifications to one queue per destination (email, irc, matrix). They would be on a
  FMN-specific vhost on the RabbitMQ server. Then write AMQP consumers for these queues
  to actually send the notifications. This would allow the IRC and Matrix notifiers to
  maintain a permanent connection to the IRC/Matrix server(s).

This work requires all applications to use Fedora Messaging Message Schemas, because the
triage component will rely on schemas to extract information from the messages.

Later on, when we refer to artifacts, we mean artifacts produced by the Fedora
infrastructure: packages, containers, modules, flatpaks.

Stakeholders
------------

- `@devel list question to gather requirements
  <https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/SGGM3XPGQX77SEDOLEI376SZVRFKTXGR/>`__
- `Fedora Discussion thread
  <https://discussion.fedoraproject.org/t/notifications-application-aka-fmn-aka-apps-fedoraproject-org-notifications/36914/2>`__
- Also ask the @mindshare / @ambassadors for the needs of non-packagers

Requirements
------------

Tracking
~~~~~~~~

- As a Fedora Contributor, I want to ask to be notified of what happens on artifacts I
  own
- As a Fedora Contributor, I want to ask to be notified of what happens on artifacts
  owned by a group I am a member of
- As a Fedora User, I want to ask to be notified of what happens on any artifact by
  entering its type and its name
- As a Fedora User, I want to be notified of events referring to my username
- As a Fedora User, I want to be notified of events referring to a group I am a member
  of (requires a `new feature
  <https://github.com/fedora-infra/fedora-messaging/issues/252>`__ in Fedora Messaging)
- As a Fedora User, I want to select which groups I want to be notified about among the
  groups I’m a member of (some groups own so many artifacts it would be too many
  notifications, eg the python maintainers group)
- As a Fedora User, I want to be able to follow someone (for example, my mentee)

Filtering
~~~~~~~~~

- As a Fedora User, I want to be able to set the minimum severity level on each rule
  (Fedora Messages have the notion of severity) (optional)
- As a Fedora User, I want to be able to block or allow notifications from a particular
  application (koji, bodhi, dist-git, etc)
- As a Fedora User, I want to be able to disable notifications for my own actions. They
  should be disabled by default.
- As a Fedora User, I want to be able to block or allow notifications from a particular
  topic. The list of known topics can be obtained from the installed schemas. This
  overrides the app allow/block list, for example I don’t want koji messages except for
  the “build complete” ones.

Destinations
~~~~~~~~~~~~

- As a Fedora User, I want to my notifications to be sent via email if I set so.
- As a Fedora User, I want to my notifications to be sent via IRC if I set so.
- As a Fedora User, I want to my notifications to be sent via Matrix if I set so
  (optional).
- As a Fedora User, I want to be able to choose one or more destination(s) on each rule
  (my objects, object foo, object bar, etc)
- As a Fedora user, I want my address for each destination to be retreived from my FAS
  account. If I have multiple values set for a destination, I want to be able to choose
  one.
- As a Fedora User, I want to be able to do filtering on my notification email based on
  application name and/or topic. A good way to offer this possibility would be to set
  these values as email headers

Deployment
~~~~~~~~~~

- As a Fedora Sysadmin, I want to be able to run multiple instance of any component of
  FMN in Openshift if I need to, for performance reasons.
- As a Fedora User, I want only one notification sent when a message matches my filters.
  Do note that during schema transition periods, the same event may produce multiple
  messages on different topics (``fedora.event.v1`` and ``fedora.event.v2``). I think it
  would be preferrable to subscribe the triage component to a fixed list of known topics
  instead of subscribing it to everything (``#``).
- As a Fedora Sysadmin, I want the Destination Handler components to be as simple and
  fast as possible because it will not be easy to have multiple instances of them (not
  sure if IRC allows multiple connections of the same account)

Misc
~~~~

- As an Application Developper, I want to set the summary and the body (where
  applicable) of what will be sent out to users in my message schema.
- As a Fedora User, I want to set my notification rules via a web UI

Suggested design
----------------

.. image:: workflow.png

Data model and configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- There are multiple Tracking Rules defined in FMN, that accept zero or more parameters.
- There are multiple supported Destinations in FMN
- There are multiple Filters defined in FMN, that accept zero or more parameters.
- Users can activate zero or more Tracking Rules and set the corresponding parameters if
  applicable.
- Users can set one or more Destinations for a Tracking Rule
- Users can set one or more Filters for a Tracking Rule and a Destination, and set the
  corresponding parameters if applicable.

Workflow
~~~~~~~~

- A Runner is created for each Tracking Rule
- Incoming messages are passed to each Runner (parallelism is possible here)
- The Runner applies the Tracking Rule over the messages, which produces a list of
  usernames to notify
- For each username:

  - check which destination(s) they want for this Tracking Rule
  - For each destination:

    - check if the incoming message’s properties pass all the Filters for this Tracking
      Rule and this Destination
    - If so, lookup the desired address in the DB
    - check that the value is still in the user’s FAS profile, or fallback to the FAS
      profile’s first value
    - Produce a Notification for this incoming message. The Notification will contain
      the destination name, the address, a subject (optional), and a body.

- Notifications produced by all the Tracking Rules are de-duplicated before being added
  to the Destination queues. One incoming message should produce at most one
  Notification for a user and a Destination.
- Destination Handlers are separate processes that will get notifications from their
  queue and send the notification to the corresponding service.

Possible tracking rules
~~~~~~~~~~~~~~~~~~~~~~~

Artifact owner
++++++++++++++

For each artifact in the message, get the artifact owners. For each owner, check in the
DB whether they have selected to be notified for their artifacts. If so, add a
notification for each selected destination if they pass filtering.

Artifact owned by one of my selected groups
+++++++++++++++++++++++++++++++++++++++++++

For each artifact in the message, get the artifact owner groups. For each groups, get
the list of members and check in the DB whether they have selected to be notified for
artifacts in this group. If so, add a notification for each selected destination if they
pass filtering.

Artifact follower
+++++++++++++++++

For each artifact in the message, get in the DB the list of users who follow those
artifacts. For each of them, add a notification for each selected destination if they
pass filtering.

Events referring to me
++++++++++++++++++++++

For each username in the message, check in the DB whether they have selected to be
notified for events referring to them. If so, add a notification for each selected
destination if they pass filtering.

Followers
+++++++++

For each username in the message, get in the DB the list of users who follow them, and
add a notification for each selected destination if they pass filtering.

Possible Filters
~~~~~~~~~~~~~~~~

The Filters should take the destination into account, to be able to filter a noisy app
on IRC but not on email for example.

Application
+++++++++++

If the application is in the user’s blocklist for the destination, and not in the user’s
allowlist: cancel the notification. The list of supported applications to present in the
UI can be extracted from the installed message schemas.

Severity
++++++++

If the severity is below the minimal severity threshold that the user has set for this
destination, cancel the notification.

My actions
++++++++++

If the message results of this user’s action, cancel the notification.

Topic
+++++

If the topic matches (*fnmatch*) any of the user’s topic blocklist entries, and does not
match any of the user’s topic allowlist entries: cancel the notification

Existing FMN
------------

Current FMN code and usage analysis
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The current FMN application implements a fedmsg based consumer, and a Flask based webui
for users to create, modify, and manage their filters.

Current Users
+++++++++++++

Currently there are 6,777 users with accounts on FMN.

Contexts
++++++++

A context in FMN is the top-level for a user to define their filters. A context is where
a message is sent after being matched. FMN in production has 2 contexts that users can
use:

- email
- irc

Each of these contexts can be enabled, which disables all the filters defined under that
context.

Of the 6,777 users in FMN, the following is a breakdown of these users’ enabled
contexts:

- Both email and irc enabled: **1048 users**
- Only email enabled: **4619 users**
- Only irc enabled: **169 users**
- No contexts enabled: **941 users**

Filters and Rules
+++++++++++++++++

For each context, a user defines zero to many filters. Each filter is made up of 1 to
many different rules which are defined in code.

Filters defaults
................

When a user logs into FMN for the first time, they are set up with a copy of the default
filters.

The current set of default filters and rules are:

- Events on packages that I own’:

  - [‘fmn.rules:user_package_filter’, {‘fasnick’: username}],
  - [‘fmn.rules:user_filter’, {‘fasnick’: username}],
  - [‘fmn.rules:koji_instance’, {‘instance’: ‘ppc,s390,arm’}],
  - [‘fmn.rules:taskotron_result_new’, {}],
  - [‘fmn.rules:git_pkgdb2branch_start’, {}],
  - [‘fmn.rules:git_pkgdb2branch_complete’, {}],
  - [‘fmn.rules:fedoratagger_catchall’, {}],
  - [‘fmn.rules:faf_report_threshold1’, {}],
  - [‘fmn.rules:faf_problem_threshold1’, {}],
  - [‘fmn.rules:bodhi_masher_start’, {}],
  - [‘fmn.rules:mdapi_repo_update’, {}],
  - [‘fmn.rules:ci_step_complete’, {}],
  - [‘fmn.rules:copr_build_start’, {}],
  - [‘fmn.rules:copr_chroot_start’, {}],
  - [‘fmn.rules:copr_worker_create’, {}],
  - [‘fmn.rules:koji_scratch_build_started’, {}],
  - [‘fmn.rules:koji_build_started’, {}],
  - [‘fmn.rules:koji_rpm_sign’, {}],
  - [‘fmn.rules:koji_tag’, {}],
  - [‘fmn.rules:koji_untag’, {}],
  - [‘fmn.rules:koji_repo_done’, {}],
  - [‘fmn.rules:koji_repo_init’, {}]
  - [‘fmn.rules:koji_package_list_change’, {}]
  - [‘fmn.rules:summershum_catchall’, {}]
  - [‘fmn.rules:anitya_catchall’, {}]]

- ‘Events referring to my username’: [[‘fmn.rules:user_filter’, {‘fasnick’: username}],]
- ‘Critical taskotron tasks on my packages’

  - [‘fmn.rules:user_package_filter’, {‘fasnick’: username}],
  - [‘fmn.rules:taskotron_release_critical_task’, {}],
  - [‘fmn.rules:taskotron_task_particular_or_changed_outcome’, {‘outcome’: ‘FAILED’}]

Because the configuration of the default setup has changed over the years, users that
logged in earlier have older default filter setups, making it harder to figure out who
is using defaults.

Current UI
++++++++++

Login page
..........

.. code-block::

    apps.fedoraproject.org/notifications/

.. image:: oldscreenshots/login.png

About page
..........

.. code-block::

    apps.fedoraproject.org/notifications/about

.. image:: oldscreenshots/about.png

Profile page
............

.. code-block::

    apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/

.. image:: oldscreenshots/profile.png

Filter list per context
.......................

.. code-block::

    https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/irc/
    https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/email/

.. image:: oldscreenshots/filterlist.png

Create / Edit Filter page
.........................

.. code-block::

    https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/irc/<filter_id>/
    https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/email/<filter_id>/

.. image:: oldscreenshots/createfilter.png

Skillsets
---------

- fedora messaging
- Web application UI
- Python (database)
- Rabbitmq (used in the backend of FMN and could be useful to know)
- Openshift
- IRC - connectivity to new systems in use (matrix)

Open Questions
--------------

- What are the core services the new FMN will need to support?
- Are there fedora-messaging schemas for each of the core services in place already?
- Does the schemas need revision, ie do they still work?
- Who would be good to get a review on UI? Mo Duffy potentially? Could this go in as a
  blog post of sorts?

Possible Timeline
-----------------

In 1 month
~~~~~~~~~~

- Email notifying community about work started on new FMN service has gone out
- The development team has been onboarded
- The backlog has epic level tickets created
- Testing criteria is loosely defined for performance, ie what is the min/max acceptable
  time for receiving a notification

In 3 months
~~~~~~~~~~~

- An update is sent to the lists on work underway
- Infra leads have been invited to a review of the work to date
- A firmer timeline for project delivery has been defined
- There is testing criteria/benchmarking agreed to

In 6 months
~~~~~~~~~~~

- The service is deployed in staging *or* its in production but it is not enabled for
  everyone yet
- Testers are invited to onboard into the new service and help test performance
- An outage period for cutover to the new service has been identified and planned if
  needed

In 9 months
~~~~~~~~~~~

- Hotfixes are applied