Infrastructure/arc
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Add draft of 2022 FMN research from the hack.md

Browse source 
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
This commit is contained in:
Ryan Lercho 2022-05-05 14:30:48 +10:00
parent 89e13e4513
commit df2453e0d4
3 changed files with 472 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

396
docs/fmn/april2022/index.rst Normal file
View file

@ -0,0 +1,396 @@
FMN Replacement (April 2022)
============================
References
----------
- `mini-initiative <https://pagure.io/cpe/initiatives-proposal/issue/10>`__
- `ARC output <https://fedora-arc.readthedocs.io/>`__
Design options
--------------
- jcline’s option: write a RabbitMQ extension. While it would probably
be way more efficient we don’t have the knowledge to write or
maintain this code. Also, it would require adding headers for the
fields we want to filter on, such as usernames and packages, and
we’ll probably loose in flexibility.
- write a fedora-messaging consumer that would triage incoming message
and produce add notifications to one queue per destination (email,
irc, matrix). They would be on a FMN-specific vhost on the RabbitMQ
server. Then write AMQP consumers for these queues to actually send
the notifications. This would allow the IRC and Matrix notifiers to
maintain a permanent connection to the IRC/Matrix server(s).
This work requires all applications to use Fedora Messaging Message
Schemas, because the triage component will rely on schemas to extract
information from the messages.
Later on, whe we refer to artifacts, we mean artifacts produced by the
Fedora infrastructure: packages, containers, modules, flatpaks.
Stakeholders
------------
- [@devel list question to gather
requirements](https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/SGGM3XPGQX77SEDOLEI376SZVRFKTXGR/)
- `Fedora Discussion
thread <https://discussion.fedoraproject.org/t/notifications-application-aka-fmn-aka-apps-fedoraproject-org-notifications/36914/2>`__
- Also ask the @mindshare / @ambassadors for the needs of non-packagers
Requirements
------------
Tracking
~~~~~~~~
- As a Fedora Contributor, I want to ask to be notified of what happens
on artifacts I own
- As a Fedora Contributor, I want to ask to be notified of what happens
on artifacts owned by a group I am a member of
- As a Fedora User, I want to ask to be notified of what happens on any
artifact by entering its type and its name
- As a Fedora User, I want to be notified of events referring to my
username
- As a Fedora User, I want to be notified of events referring to a
group I am a member of (requires a `new
feature <https://github.com/fedora-infra/fedora-messaging/issues/252>`__
in Fedora Messaging)
- As a Fedora User, I want to select which groups I want to be notified
about among the groups I’m a member of (some groups own so many
artifacts it would be too many notifications, eg the python
maintainers group)
- As a Fedora User, I want to be able to follow someone (for example,
my mentee)
Filtering
~~~~~~~~~
- As a Fedora User, I want to be able to set the minimum severity level
on each rule (Fedora Messages have the notion of severity) (optional)
- As a Fedora User, I want to be able to block or allow notifications
from a particular application (koji, bodhi, dist-git, etc)
- As a Fedora User, I want to be able to disable notifications for my
own actions. They should be disabled by default.
- As a Fedora User, I want to be able to block or allow notifications
from a particular topic. The list of known topics can be obtained
from the installed schemas. This overrides the app allow/block list,
for example I don’t want koji messages except for the “build
complete” ones.
Destinations
~~~~~~~~~~~~
- As a Fedora User, I want to my notifications to be sent via email if
I set so.
- As a Fedora User, I want to my notifications to be sent via IRC if I
set so.
- As a Fedora User, I want to my notifications to be sent via Matrix if
I set so (optional).
- As a Fedora User, I want to be able to choose one or more
destination(s) on each rule (my objects, object foo, object bar, etc)
- As a Fedora user, I want my address for each destination to be
retreived from my FAS account. If I have multiple values set for a
destination, I want to be able to choose one.
- As a Fedora User, I want to be able to do filtering on my
notification email based on application name and/or topic. A good way
to offer this possibility would be to set these values as email
headers
Deployment
~~~~~~~~~~
- As a Fedora Sysadmin, I want to be able to run multiple instance of
any component of FMN in Openshift if I need to, for performance
reasons.
- As a Fedora User, I want only one notification sent when a message
matches my filters. Do note that during schema transition periods,
the same event may produce multiple messages on different topics
(``fedora.event.v1`` and ``fedora.event.v2``). I think it would be
preferrable to subscribe the triage component to a fixed list of
known topics instead of subscribing it to everything (``#``).
- As a Fedora Sysadmin, I want the Destination Handler components to be
as simple and fast as possible because it will not be easy to have
multiple instances of them (not sure if IRC allows multiple
connections of the same account)
Misc
~~~~
- As an Application Developper, I want to set the summary and the body
(where applicable) of what will be sent out to users in my message
schema.
- As a Fedora User, I want to set my notification rules via a web UI
Suggested design
----------------
.. image:: https://i.imgur.com/SsqFphF.png
(`graph
source <https://edotor.net/?engine=dot#strict%20digraph%20%7B%0A%20%20%20%20incoming%20%5Blabel%3D%22Incoming%20Message%22%5D%3B%0A%20%20%20%20t1%20%5Blabel%3D%22Tracking%20Rule%201%22%5D%3B%0A%20%20%20%20t2%20%5Blabel%3D%22Tracking%20Rule%202%22%5D%3B%0A%20%20%20%20t3%20%5Blabel%3D%22Tracking%20Rule%203%22%5D%3B%0A%20%20%20%20f1%20%5Blabel%3D%22Filter%201%22%5D%0A%20%20%20%20f2%20%5Blabel%3D%22Filter%202%22%5D%0A%20%20%20%20d1%20%5Blabel%3D%22Destination%201%22%5D%0A%20%20%20%20d2%20%5Blabel%3D%22Destination%202%22%5D%0A%20%20%20%20notif%20%5Blabel%3D%22Notification%22%5D%0A%20%20%20%20incoming%20-%3E%20t1%3B%0A%20%20%20%20incoming%20-%3E%20t2%3B%0A%20%20%20%20incoming%20-%3E%20t3%3B%0A%20%20%20%20t1%20-%3E%20d1%3B%0A%20%20%20%20t1%20-%3E%20d2%3B%0A%20%20%20%20d1%20-%3E%20f1%3B%0A%20%20%20%20d1%20-%3E%20f2%3B%0A%20%20%20%20f1%20-%3E%20notif%3B%0A%20%20%20%20f2%20-%3E%20notif%3B%0A%7D%0A>`__)
Data model and configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- There are multiple Tracking Rules defined in FMN, that accept zero or
more parameters.
- There are multiple supported Destinations in FMN
- There are multiple Filters defined in FMN, that accept zero or more
parameters.
- Users can activate zero or more Tracking Rules and set the
corresponding parameters if applicable.
- Users can set one or more Destinations for a Tracking Rule
- Users can set one or more Filters for a Tracking Rule and a
Destination, and set the corresponding parameters if applicable.
Workflow
~~~~~~~~
- A Runner is created for each Tracking Rule
- Incoming messages are passed to each Runner (parallelism is possible
here)
- The Runner applies the Tracking Rule over the messages, which
produces a list of usernames to notify
- For each username:
- check which destination(s) they want for this Tracking Rule
- For each destination:
- check if the incoming message’s properties pass all the Filters
for this Tracking Rule and this Destination
- If so, lookup the desired address in the DB
- check that the value is still in the user’s FAS profile, or
fallback to the FAS profile’s first value
- Produce a Notification for this incoming message. The
Notification will contain the destination name, the address, a
subject (optional), and a body.
- Notifications produced by all the Tracking Rules are de-duplicated
before being added to the Destination queues. One incoming message
should produce at most one Notification for a user and a Destination.
- Destination Handlers are separate processes that will get
notifications from their queue and send the notification to the
corresponding service.
Possible tracking rules
~~~~~~~~~~~~~~~~~~~~~~~
Artifact owner
^^^^^^^^^^^^^^
For each artifact in the message, get the artifact owners. For each
owner, check in the DB whether they have selected to be notified for
their artifacts. If so, add a notification for each selected destination
if they pass filtering.
Artifact owned by one of my selected groups
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For each artifact in the message, get the artifact owner groups. For
each groups, get the list of members and check in the DB whether they
have selected to be notified for artifacts in this group. If so, add a
notification for each selected destination if they pass filtering.
Artifact follower
^^^^^^^^^^^^^^^^^
For each artifact in the message, get in the DB the list of users who
follow those artifacts. For each of them, add a notification for each
selected destination if they pass filtering.
Events referring to me
^^^^^^^^^^^^^^^^^^^^^^
For each username in the message, check in the DB whether they have
selected to be notified for events referring to them. If so, add a
notification for each selected destination if they pass filtering.
Followers
^^^^^^^^^
For each username in the message, get in the DB the list of users who
follow them, and add a notification for each selected destination if
they pass filtering.
Possible Filters
~~~~~~~~~~~~~~~~
The Filters should take the destination into account, to be able to
filter a noisy app on IRC but not on email for example.
Application
^^^^^^^^^^^
If the application is in the user’s blocklist for the destination, and
not in the user’s allowlist: cancel the notification. The list of
supported applications to present in the UI can be extracted from the
installed message schemas.
Severity
^^^^^^^^
If the severity is below the minimal severity threshold that the user
has set for this destination, cancel the notification.
My actions
^^^^^^^^^^
If the message results of this user’s action, cancel the notification.
Topic
^^^^^
If the topic matches (*fnmatch*) any of the user’s topic blocklist
entries, and does not match any of the user’s topic allowlist entries:
cancel the notification
Existing FMN
------------
Current Issues
~~~~~~~~~~~~~~
1. **message delivery lag** in times of peak congestion on the message
queue (e.g. during a mass rebuild) it can take **days** for messages
to be delivered to users.
2. **fedmsg**
3. **python2**
4. **UI** after adhoc discussions with the development community one of
the major issues is the fmn UI – creating new filters is cumbersome,
and most people keep the standard filters or just turn them off.
Current FMN code and usage analysis
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The current FMN application implements a fedmsg based consumer, and a
Flask based webui for users to create, modify, and manage their filters.
Current Users
^^^^^^^^^^^^^
Currently there are 6,777 users with accounts on FMN.
Contexts
^^^^^^^^
A context in FMN is the top-level for a user to define their filters. A
context is where a message is sent after being matched. FMN in
production has 2 contexts that users can use:
- email
- irc
Each of these contexts can be enabled, which disables all the filters
defined under that context.
Of the 6,777 users in FMN, the following is a breakdown of these users’
enabled contexts:
- Both email and irc enabled: **1048 users**
- Only email enabled: **4619 users**
- Only irc enabled: **169 users**
- No contexts enabled: **941 users**
Filters and Rules
^^^^^^^^^^^^^^^^^
For each context, a user defines zero to many filters. Each filter is
made up of 1 to many different rules which are defined in code.
Filters defaults
''''''''''''''''
When a user logs into FMN for the first time, they are set up with a
copy of the default filters.
The current set of default filters and rules are:
- Events on packages that I own’:
- [‘fmn.rules:user_package_filter’, {‘fasnick’: username}],
- [‘fmn.rules:user_filter’, {‘fasnick’: username}],
- [‘fmn.rules:koji_instance’, {‘instance’: ‘ppc,s390,arm’}],
- [‘fmn.rules:taskotron_result_new’, {}],
- [‘fmn.rules:git_pkgdb2branch_start’, {}],
- [‘fmn.rules:git_pkgdb2branch_complete’, {}],
- [‘fmn.rules:fedoratagger_catchall’, {}],
- [‘fmn.rules:faf_report_threshold1’, {}],
- [‘fmn.rules:faf_problem_threshold1’, {}],
- [‘fmn.rules:bodhi_masher_start’, {}],
- [‘fmn.rules:mdapi_repo_update’, {}],
- [‘fmn.rules:ci_step_complete’, {}],
- [‘fmn.rules:copr_build_start’, {}],
- [‘fmn.rules:copr_chroot_start’, {}],
- [‘fmn.rules:copr_worker_create’, {}],
- [‘fmn.rules:koji_scratch_build_started’, {}],
- [‘fmn.rules:koji_build_started’, {}],
- [‘fmn.rules:koji_rpm_sign’, {}],
- [‘fmn.rules:koji_tag’, {}],
- [‘fmn.rules:koji_untag’, {}],
- [‘fmn.rules:koji_repo_done’, {}],
- [‘fmn.rules:koji_repo_init’, {}]
- [‘fmn.rules:koji_package_list_change’, {}]
- [‘fmn.rules:summershum_catchall’, {}]
- [‘fmn.rules:anitya_catchall’, {}]]
- ‘Events referring to my username’: [[‘fmn.rules:user_filter’,
{‘fasnick’: username}],]
- ‘Critical taskotron tasks on my packages’
- [‘fmn.rules:user_package_filter’, {‘fasnick’: username}],
- [‘fmn.rules:taskotron_release_critical_task’, {}],
- [‘fmn.rules:taskotron_task_particular_or_changed_outcome’,
{‘outcome’: ‘FAILED’}]
Because the configuration of the default setup has changed over the
years, users that logged in earlier have older default filter setups,
making it harder to figure out who is using defaults.
Skillsets
---------
* fedora messaging
* Web application UI
* Python (database)
* Rabbitmq (used in the backend of FMN and could be useful to know)
* Openshift
* IRC - connectivity to new systems in use (matrix)
Open Questions
--------------
* What are the core services the new FMN will need to support?
* Are there fedora-messaging schemas for each of the core services in place already?
* Does the schemas need revision, ie do they still work?
* Who would be good to get a review on UI? Mo Duffy potentially? Could this go in as a blog post of sorts?
Possible Timeline
^^^^^^^^^^^^^^^^^
In 1 month
----------
* Email notifying community about work started on new FMN service has gone out
* The development team has been onboarded
* The backlog has epic level tickets created
* Testing criteria is loosely defined for performance, ie what is the min/max acceptable time for receiving a notification
In 3 months
-----------
* An update is sent to the lists on work underway
* Infra leads have been invited to a review of the work to date
* A firmer timeline for project delivery has been defined
* There is testing criteria/benchmarking agreed to
In 6 months
-----------
* The service is deployed in staging *or* its in production but it is not enabled for everyone yet
* Testers are invited to onboard into the new service and help test performance
* An outage period for cutover to the new service has been identified and planned if needed
In 9 months
-----------
* Hotfixes are applied

74
docs/fmn/index.rst
View file

@ -1,70 +1,8 @@
FMN replacement research FMN
======================== ===========
`FMN <https://apps.fedoraproject.org/notifications/>`_ is application where users with fedora account .. toctree::
setup notifications to be sent on fedora-messaging messages that match a specific filter. :maxdepth: 3
Our current deployment running on https://apps.fedoraproject.org/notifications/. june2021/index
april2022/index
It uses https://github.com/fedora-infra/fmn
The software is currently under our fedora-infra organization.
Abstract take-away
------------------
FMN is a project that allows people in our community to get notified when messages that interest them
fire on the message-bus, making the message-bus more useful to people that are not
directly developing or troubleshooting applications running in our infra.
This means we really don't want to drop it's support.
A two person team with skill in python development, with access to current FMN infrastructure,
and ability to run deployment playbooks should be able to finish
the work started in Pierre's POC and end up with up-to-date and more maintainable project.
Solutions to be explored
------------------------
We have three options:
- Upgrade FMN to supported version of Python and invest in the maintenance
- Create a new bespoke application and invest in the maintenanece
- Find off-the shelf application that would fill the role without us needing to maintain it
Currently we are proposing primarily the first option.
Preliminary notes on maintaining current FMN
--------------------------------------------
Based on Pierre's POC, FMN is upgrade-able to recent versions
of python and its libraries.
This suggests it is prudent to invest in it further
- FMN is key piece to fullfil the promise of having a message-bus useful to our community
- green-field development is more costly and harder to ramp up
Outstanding issues to be tackled by the initiative:
- verify the upgrades actually work by thorough testing
- migrate FMN to the fedora-messaging message-bus, as it still relies on fedmsg
- improve it's development state, by improving the test coverage, adding more static-checks e.t.c.
Preliminary notes on using other off-the-shelf applications
-----------------------------------------------------------
As we want people to be able to get notified on messages on the bus,
we'd need to integrate with our fedora-messaging infrastructure and our
account system. We didn't find application that could be deployed and
fullfil all of the needs FMN currently serves
Create a new bespoke application that better suits our needs
------------------------------------------------------------
This is an option we'd take if and only if the path of maintaining the original application
was not feasible, or there was some way to greatly simplify the new implementation.
With FMN neither is the case.

70
docs/fmn/june2021/index.rst Normal file
View file

@ -0,0 +1,70 @@
FMN replace or fix research (June 2021)
=======================================
`FMN <https://apps.fedoraproject.org/notifications/>`_ is application where users with fedora account
setup notifications to be sent on fedora-messaging messages that match a specific filter.
Our current deployment running on https://apps.fedoraproject.org/notifications/.
It uses https://github.com/fedora-infra/fmn
The software is currently under our fedora-infra organization.
Abstract take-away
------------------
FMN is a project that allows people in our community to get notified when messages that interest them
fire on the message-bus, making the message-bus more useful to people that are not
directly developing or troubleshooting applications running in our infra.
This means we really don't want to drop it's support.
A two person team with skill in python development, with access to current FMN infrastructure,
and ability to run deployment playbooks should be able to finish
the work started in Pierre's POC and end up with up-to-date and more maintainable project.
Solutions to be explored
------------------------
We have three options:
- Upgrade FMN to supported version of Python and invest in the maintenance
- Create a new bespoke application and invest in the maintenanece
- Find off-the shelf application that would fill the role without us needing to maintain it
Currently we are proposing primarily the first option.
Preliminary notes on maintaining current FMN
--------------------------------------------
Based on Pierre's POC, FMN is upgrade-able to recent versions
of python and its libraries.
This suggests it is prudent to invest in it further
- FMN is key piece to fullfil the promise of having a message-bus useful to our community
- green-field development is more costly and harder to ramp up
Outstanding issues to be tackled by the initiative:
- verify the upgrades actually work by thorough testing
- migrate FMN to the fedora-messaging message-bus, as it still relies on fedmsg
- improve it's development state, by improving the test coverage, adding more static-checks e.t.c.
Preliminary notes on using other off-the-shelf applications
-----------------------------------------------------------
As we want people to be able to get notified on messages on the bus,
we'd need to integrate with our fedora-messaging infrastructure and our
account system. We didn't find application that could be deployed and
fullfil all of the needs FMN currently serves
Create a new bespoke application that better suits our needs
------------------------------------------------------------
This is an option we'd take if and only if the path of maintaining the original application
was not feasible, or there was some way to greatly simplify the new implementation.
With FMN neither is the case.