From df2453e0d4d587e164c1e3f4462015bfcf67037f Mon Sep 17 00:00:00 2001 From: Ryan Lerch Date: Thu, 5 May 2022 14:30:48 +1000 Subject: [PATCH] Add draft of 2022 FMN research from the hack.md Signed-off-by: Ryan Lerch --- docs/fmn/april2022/index.rst | 396 +++++++++++++++++++++++++++++++++++ docs/fmn/index.rst | 74 +------ docs/fmn/june2021/index.rst | 70 +++++++ 3 files changed, 472 insertions(+), 68 deletions(-) create mode 100644 docs/fmn/april2022/index.rst create mode 100644 docs/fmn/june2021/index.rst diff --git a/docs/fmn/april2022/index.rst b/docs/fmn/april2022/index.rst new file mode 100644 index 0000000..670ab2d --- /dev/null +++ b/docs/fmn/april2022/index.rst @@ -0,0 +1,396 @@ +FMN Replacement (April 2022) +============================ + +References +---------- + +- `mini-initiative `__ +- `ARC output `__ + +Design options +-------------- + +- jcline’s option: write a RabbitMQ extension. While it would probably + be way more efficient we don’t have the knowledge to write or + maintain this code. Also, it would require adding headers for the + fields we want to filter on, such as usernames and packages, and + we’ll probably loose in flexibility. +- write a fedora-messaging consumer that would triage incoming message + and produce add notifications to one queue per destination (email, + irc, matrix). They would be on a FMN-specific vhost on the RabbitMQ + server. Then write AMQP consumers for these queues to actually send + the notifications. This would allow the IRC and Matrix notifiers to + maintain a permanent connection to the IRC/Matrix server(s). + +This work requires all applications to use Fedora Messaging Message +Schemas, because the triage component will rely on schemas to extract +information from the messages. + +Later on, whe we refer to artifacts, we mean artifacts produced by the +Fedora infrastructure: packages, containers, modules, flatpaks. + +Stakeholders +------------ + +- [@devel list question to gather + requirements](https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/SGGM3XPGQX77SEDOLEI376SZVRFKTXGR/) +- `Fedora Discussion + thread `__ +- Also ask the @mindshare / @ambassadors for the needs of non-packagers + +Requirements +------------ + +Tracking +~~~~~~~~ + +- As a Fedora Contributor, I want to ask to be notified of what happens + on artifacts I own +- As a Fedora Contributor, I want to ask to be notified of what happens + on artifacts owned by a group I am a member of +- As a Fedora User, I want to ask to be notified of what happens on any + artifact by entering its type and its name +- As a Fedora User, I want to be notified of events referring to my + username +- As a Fedora User, I want to be notified of events referring to a + group I am a member of (requires a `new + feature `__ + in Fedora Messaging) +- As a Fedora User, I want to select which groups I want to be notified + about among the groups I’m a member of (some groups own so many + artifacts it would be too many notifications, eg the python + maintainers group) +- As a Fedora User, I want to be able to follow someone (for example, + my mentee) + +Filtering +~~~~~~~~~ + +- As a Fedora User, I want to be able to set the minimum severity level + on each rule (Fedora Messages have the notion of severity) (optional) +- As a Fedora User, I want to be able to block or allow notifications + from a particular application (koji, bodhi, dist-git, etc) +- As a Fedora User, I want to be able to disable notifications for my + own actions. They should be disabled by default. +- As a Fedora User, I want to be able to block or allow notifications + from a particular topic. The list of known topics can be obtained + from the installed schemas. This overrides the app allow/block list, + for example I don’t want koji messages except for the “build + complete” ones. + +Destinations +~~~~~~~~~~~~ + +- As a Fedora User, I want to my notifications to be sent via email if + I set so. +- As a Fedora User, I want to my notifications to be sent via IRC if I + set so. +- As a Fedora User, I want to my notifications to be sent via Matrix if + I set so (optional). +- As a Fedora User, I want to be able to choose one or more + destination(s) on each rule (my objects, object foo, object bar, etc) +- As a Fedora user, I want my address for each destination to be + retreived from my FAS account. If I have multiple values set for a + destination, I want to be able to choose one. +- As a Fedora User, I want to be able to do filtering on my + notification email based on application name and/or topic. A good way + to offer this possibility would be to set these values as email + headers + +Deployment +~~~~~~~~~~ + +- As a Fedora Sysadmin, I want to be able to run multiple instance of + any component of FMN in Openshift if I need to, for performance + reasons. +- As a Fedora User, I want only one notification sent when a message + matches my filters. Do note that during schema transition periods, + the same event may produce multiple messages on different topics + (``fedora.event.v1`` and ``fedora.event.v2``). I think it would be + preferrable to subscribe the triage component to a fixed list of + known topics instead of subscribing it to everything (``#``). +- As a Fedora Sysadmin, I want the Destination Handler components to be + as simple and fast as possible because it will not be easy to have + multiple instances of them (not sure if IRC allows multiple + connections of the same account) + +Misc +~~~~ + +- As an Application Developper, I want to set the summary and the body + (where applicable) of what will be sent out to users in my message + schema. +- As a Fedora User, I want to set my notification rules via a web UI + +Suggested design +---------------- + +.. image:: https://i.imgur.com/SsqFphF.png + +(`graph +source `__) + +Data model and configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- There are multiple Tracking Rules defined in FMN, that accept zero or + more parameters. +- There are multiple supported Destinations in FMN +- There are multiple Filters defined in FMN, that accept zero or more + parameters. +- Users can activate zero or more Tracking Rules and set the + corresponding parameters if applicable. +- Users can set one or more Destinations for a Tracking Rule +- Users can set one or more Filters for a Tracking Rule and a + Destination, and set the corresponding parameters if applicable. + +Workflow +~~~~~~~~ + +- A Runner is created for each Tracking Rule +- Incoming messages are passed to each Runner (parallelism is possible + here) +- The Runner applies the Tracking Rule over the messages, which + produces a list of usernames to notify +- For each username: + + - check which destination(s) they want for this Tracking Rule + - For each destination: + + - check if the incoming message’s properties pass all the Filters + for this Tracking Rule and this Destination + - If so, lookup the desired address in the DB + - check that the value is still in the user’s FAS profile, or + fallback to the FAS profile’s first value + - Produce a Notification for this incoming message. The + Notification will contain the destination name, the address, a + subject (optional), and a body. + +- Notifications produced by all the Tracking Rules are de-duplicated + before being added to the Destination queues. One incoming message + should produce at most one Notification for a user and a Destination. +- Destination Handlers are separate processes that will get + notifications from their queue and send the notification to the + corresponding service. + +Possible tracking rules +~~~~~~~~~~~~~~~~~~~~~~~ + +Artifact owner +^^^^^^^^^^^^^^ + +For each artifact in the message, get the artifact owners. For each +owner, check in the DB whether they have selected to be notified for +their artifacts. If so, add a notification for each selected destination +if they pass filtering. + +Artifact owned by one of my selected groups +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For each artifact in the message, get the artifact owner groups. For +each groups, get the list of members and check in the DB whether they +have selected to be notified for artifacts in this group. If so, add a +notification for each selected destination if they pass filtering. + +Artifact follower +^^^^^^^^^^^^^^^^^ + +For each artifact in the message, get in the DB the list of users who +follow those artifacts. For each of them, add a notification for each +selected destination if they pass filtering. + +Events referring to me +^^^^^^^^^^^^^^^^^^^^^^ + +For each username in the message, check in the DB whether they have +selected to be notified for events referring to them. If so, add a +notification for each selected destination if they pass filtering. + +Followers +^^^^^^^^^ + +For each username in the message, get in the DB the list of users who +follow them, and add a notification for each selected destination if +they pass filtering. + +Possible Filters +~~~~~~~~~~~~~~~~ + +The Filters should take the destination into account, to be able to +filter a noisy app on IRC but not on email for example. + +Application +^^^^^^^^^^^ + +If the application is in the user’s blocklist for the destination, and +not in the user’s allowlist: cancel the notification. The list of +supported applications to present in the UI can be extracted from the +installed message schemas. + +Severity +^^^^^^^^ + +If the severity is below the minimal severity threshold that the user +has set for this destination, cancel the notification. + +My actions +^^^^^^^^^^ + +If the message results of this user’s action, cancel the notification. + +Topic +^^^^^ + +If the topic matches (*fnmatch*) any of the user’s topic blocklist +entries, and does not match any of the user’s topic allowlist entries: +cancel the notification + +Existing FMN +------------ + +Current Issues +~~~~~~~~~~~~~~ + +1. **message delivery lag** in times of peak congestion on the message + queue (e.g. during a mass rebuild) it can take **days** for messages + to be delivered to users. +2. **fedmsg** +3. **python2** +4. **UI** after adhoc discussions with the development community one of + the major issues is the fmn UI – creating new filters is cumbersome, + and most people keep the standard filters or just turn them off. + +Current FMN code and usage analysis +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The current FMN application implements a fedmsg based consumer, and a +Flask based webui for users to create, modify, and manage their filters. + +Current Users +^^^^^^^^^^^^^ + +Currently there are 6,777 users with accounts on FMN. + +Contexts +^^^^^^^^ + +A context in FMN is the top-level for a user to define their filters. A +context is where a message is sent after being matched. FMN in +production has 2 contexts that users can use: + +- email +- irc + +Each of these contexts can be enabled, which disables all the filters +defined under that context. + +Of the 6,777 users in FMN, the following is a breakdown of these users’ +enabled contexts: + +- Both email and irc enabled: **1048 users** +- Only email enabled: **4619 users** +- Only irc enabled: **169 users** +- No contexts enabled: **941 users** + +Filters and Rules +^^^^^^^^^^^^^^^^^ + +For each context, a user defines zero to many filters. Each filter is +made up of 1 to many different rules which are defined in code. + +Filters defaults +'''''''''''''''' + +When a user logs into FMN for the first time, they are set up with a +copy of the default filters. + +The current set of default filters and rules are: + +- Events on packages that I own’: + + - [‘fmn.rules:user_package_filter’, {‘fasnick’: username}], + - [‘fmn.rules:user_filter’, {‘fasnick’: username}], + - [‘fmn.rules:koji_instance’, {‘instance’: ‘ppc,s390,arm’}], + - [‘fmn.rules:taskotron_result_new’, {}], + - [‘fmn.rules:git_pkgdb2branch_start’, {}], + - [‘fmn.rules:git_pkgdb2branch_complete’, {}], + - [‘fmn.rules:fedoratagger_catchall’, {}], + - [‘fmn.rules:faf_report_threshold1’, {}], + - [‘fmn.rules:faf_problem_threshold1’, {}], + - [‘fmn.rules:bodhi_masher_start’, {}], + - [‘fmn.rules:mdapi_repo_update’, {}], + - [‘fmn.rules:ci_step_complete’, {}], + - [‘fmn.rules:copr_build_start’, {}], + - [‘fmn.rules:copr_chroot_start’, {}], + - [‘fmn.rules:copr_worker_create’, {}], + - [‘fmn.rules:koji_scratch_build_started’, {}], + - [‘fmn.rules:koji_build_started’, {}], + - [‘fmn.rules:koji_rpm_sign’, {}], + - [‘fmn.rules:koji_tag’, {}], + - [‘fmn.rules:koji_untag’, {}], + - [‘fmn.rules:koji_repo_done’, {}], + - [‘fmn.rules:koji_repo_init’, {}] + - [‘fmn.rules:koji_package_list_change’, {}] + - [‘fmn.rules:summershum_catchall’, {}] + - [‘fmn.rules:anitya_catchall’, {}]] + +- ‘Events referring to my username’: [[‘fmn.rules:user_filter’, + {‘fasnick’: username}],] +- ‘Critical taskotron tasks on my packages’ + + - [‘fmn.rules:user_package_filter’, {‘fasnick’: username}], + - [‘fmn.rules:taskotron_release_critical_task’, {}], + - [‘fmn.rules:taskotron_task_particular_or_changed_outcome’, + {‘outcome’: ‘FAILED’}] + +Because the configuration of the default setup has changed over the +years, users that logged in earlier have older default filter setups, +making it harder to figure out who is using defaults. + +Skillsets +--------- + +* fedora messaging +* Web application UI +* Python (database) +* Rabbitmq (used in the backend of FMN and could be useful to know) +* Openshift +* IRC - connectivity to new systems in use (matrix) + +Open Questions +-------------- + +* What are the core services the new FMN will need to support? +* Are there fedora-messaging schemas for each of the core services in place already? +* Does the schemas need revision, ie do they still work? +* Who would be good to get a review on UI? Mo Duffy potentially? Could this go in as a blog post of sorts? + +Possible Timeline +^^^^^^^^^^^^^^^^^ + +In 1 month +---------- +* Email notifying community about work started on new FMN service has gone out +* The development team has been onboarded +* The backlog has epic level tickets created +* Testing criteria is loosely defined for performance, ie what is the min/max acceptable time for receiving a notification + +In 3 months +----------- + +* An update is sent to the lists on work underway +* Infra leads have been invited to a review of the work to date +* A firmer timeline for project delivery has been defined +* There is testing criteria/benchmarking agreed to + +In 6 months +----------- + +* The service is deployed in staging *or* its in production but it is not enabled for everyone yet +* Testers are invited to onboard into the new service and help test performance +* An outage period for cutover to the new service has been identified and planned if needed + +In 9 months +----------- + +* Hotfixes are applied diff --git a/docs/fmn/index.rst b/docs/fmn/index.rst index 06b6f5c..ebebe1d 100644 --- a/docs/fmn/index.rst +++ b/docs/fmn/index.rst @@ -1,70 +1,8 @@ -FMN replacement research -======================== +FMN +=========== -`FMN `_ is application where users with fedora account -setup notifications to be sent on fedora-messaging messages that match a specific filter. +.. toctree:: + :maxdepth: 3 -Our current deployment running on https://apps.fedoraproject.org/notifications/. - -It uses https://github.com/fedora-infra/fmn - -The software is currently under our fedora-infra organization. - -Abstract take-away ------------------- - -FMN is a project that allows people in our community to get notified when messages that interest them -fire on the message-bus, making the message-bus more useful to people that are not -directly developing or troubleshooting applications running in our infra. - -This means we really don't want to drop it's support. - -A two person team with skill in python development, with access to current FMN infrastructure, -and ability to run deployment playbooks should be able to finish -the work started in Pierre's POC and end up with up-to-date and more maintainable project. - -Solutions to be explored ------------------------- - -We have three options: - -- Upgrade FMN to supported version of Python and invest in the maintenance -- Create a new bespoke application and invest in the maintenanece -- Find off-the shelf application that would fill the role without us needing to maintain it - -Currently we are proposing primarily the first option. - -Preliminary notes on maintaining current FMN --------------------------------------------- - -Based on Pierre's POC, FMN is upgrade-able to recent versions -of python and its libraries. - -This suggests it is prudent to invest in it further - -- FMN is key piece to fullfil the promise of having a message-bus useful to our community -- green-field development is more costly and harder to ramp up - -Outstanding issues to be tackled by the initiative: - -- verify the upgrades actually work by thorough testing -- migrate FMN to the fedora-messaging message-bus, as it still relies on fedmsg -- improve it's development state, by improving the test coverage, adding more static-checks e.t.c. - - -Preliminary notes on using other off-the-shelf applications ------------------------------------------------------------ - -As we want people to be able to get notified on messages on the bus, -we'd need to integrate with our fedora-messaging infrastructure and our -account system. We didn't find application that could be deployed and -fullfil all of the needs FMN currently serves - - -Create a new bespoke application that better suits our needs ------------------------------------------------------------- - -This is an option we'd take if and only if the path of maintaining the original application -was not feasible, or there was some way to greatly simplify the new implementation. - -With FMN neither is the case. + june2021/index + april2022/index diff --git a/docs/fmn/june2021/index.rst b/docs/fmn/june2021/index.rst new file mode 100644 index 0000000..6f740e2 --- /dev/null +++ b/docs/fmn/june2021/index.rst @@ -0,0 +1,70 @@ +FMN replace or fix research (June 2021) +======================================= + +`FMN `_ is application where users with fedora account +setup notifications to be sent on fedora-messaging messages that match a specific filter. + +Our current deployment running on https://apps.fedoraproject.org/notifications/. + +It uses https://github.com/fedora-infra/fmn + +The software is currently under our fedora-infra organization. + +Abstract take-away +------------------ + +FMN is a project that allows people in our community to get notified when messages that interest them +fire on the message-bus, making the message-bus more useful to people that are not +directly developing or troubleshooting applications running in our infra. + +This means we really don't want to drop it's support. + +A two person team with skill in python development, with access to current FMN infrastructure, +and ability to run deployment playbooks should be able to finish +the work started in Pierre's POC and end up with up-to-date and more maintainable project. + +Solutions to be explored +------------------------ + +We have three options: + +- Upgrade FMN to supported version of Python and invest in the maintenance +- Create a new bespoke application and invest in the maintenanece +- Find off-the shelf application that would fill the role without us needing to maintain it + +Currently we are proposing primarily the first option. + +Preliminary notes on maintaining current FMN +-------------------------------------------- + +Based on Pierre's POC, FMN is upgrade-able to recent versions +of python and its libraries. + +This suggests it is prudent to invest in it further + +- FMN is key piece to fullfil the promise of having a message-bus useful to our community +- green-field development is more costly and harder to ramp up + +Outstanding issues to be tackled by the initiative: + +- verify the upgrades actually work by thorough testing +- migrate FMN to the fedora-messaging message-bus, as it still relies on fedmsg +- improve it's development state, by improving the test coverage, adding more static-checks e.t.c. + + +Preliminary notes on using other off-the-shelf applications +----------------------------------------------------------- + +As we want people to be able to get notified on messages on the bus, +we'd need to integrate with our fedora-messaging infrastructure and our +account system. We didn't find application that could be deployed and +fullfil all of the needs FMN currently serves + + +Create a new bespoke application that better suits our needs +------------------------------------------------------------ + +This is an option we'd take if and only if the path of maintaining the original application +was not feasible, or there was some way to greatly simplify the new implementation. + +With FMN neither is the case.