From 9b7c14dd1247811659be8cd8c25eabca213b3fc7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Michal=20Kone=C4=8Dn=C3=BD?= Date: Wed, 9 Nov 2022 15:30:12 +0100 Subject: [PATCH] Add webhook2fedmsg investigation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This commit contains investigation for https://pagure.io/cpe/initiatives-proposal/issue/18 It describes a solution for new app converting webhooks to fedora messaging messages. Originally this was done as part of github2fedmsg investigation, but it is a completely different proposal. Signed-off-by: Michal Konečný --- docs/github2fedmsg/index.rst | 1 - docs/github2fedmsg/webhook2fedmsg.rst | 29 ----- docs/initiatives.rst | 1 + .../design_of_webhook2fedmsg.rst | 97 +++++++++++++++++ docs/webhook2fedmsg/index.rst | 100 ++++++++++++++++++ 5 files changed, 198 insertions(+), 30 deletions(-) delete mode 100644 docs/github2fedmsg/webhook2fedmsg.rst create mode 100644 docs/webhook2fedmsg/design_of_webhook2fedmsg.rst create mode 100644 docs/webhook2fedmsg/index.rst diff --git a/docs/github2fedmsg/index.rst b/docs/github2fedmsg/index.rst index b938af9..e5cd493 100644 --- a/docs/github2fedmsg/index.rst +++ b/docs/github2fedmsg/index.rst @@ -55,7 +55,6 @@ Here is the list of ideas/things we discussed/looked at: revitalization rewrite - webhook2fedmsg enhancement_of_repository diff --git a/docs/github2fedmsg/webhook2fedmsg.rst b/docs/github2fedmsg/webhook2fedmsg.rst deleted file mode 100644 index fbe8c14..0000000 --- a/docs/github2fedmsg/webhook2fedmsg.rst +++ /dev/null @@ -1,29 +0,0 @@ -.. _webhook2fedmsg: - -webhook2fedmsg -============== - -This document will investigate the possibility to create a more general solution -converting various webhooks to fedora messaging messages. This will completely -change the github2fedmsg to modular application supporting multiple sources. - -Requirements ------------- - - -New design ----------- - - -The Good Points ---------------- - -1. Could support multiple sources in future -2. Easier to maintain -3. Easier to customize - -The Bad points --------------- - -1. More work for us -2. Existing code will be thrown away diff --git a/docs/initiatives.rst b/docs/initiatives.rst index 05a19e7..e6d731c 100644 --- a/docs/initiatives.rst +++ b/docs/initiatives.rst @@ -23,6 +23,7 @@ Completed review flask-oidc/index communishift/index kerneltest/index + webhook2fedmsg/index Implemented ----------- diff --git a/docs/webhook2fedmsg/design_of_webhook2fedmsg.rst b/docs/webhook2fedmsg/design_of_webhook2fedmsg.rst new file mode 100644 index 0000000..9eb8a5c --- /dev/null +++ b/docs/webhook2fedmsg/design_of_webhook2fedmsg.rst @@ -0,0 +1,97 @@ +.. _design_of_webhook2fedmsg: + +Design document for webhook2fedmsg +================================== + +This document will investigate the possibility to create a more general solution +converting various webhooks to fedora messaging messages. + + +External services +----------------- + +External services webhook2fedmsg will communicate with. + +* Database + + Will be used to store data about FAS user and tokens they added. We can consider supporting multiple tokens for each service per user. + For database it would be best to use existing `PostreSQL `_ + instance we already host in Fedora infra. + +* Webhook services + + The application will listen for webhooks from supported services. For each service should be separate backend, so it's easier to + remove/add new services. + +* FAS + + We need to authenticate user against Fedora Account System. + For authentication we can use `flask-oidc `_ or `authlib `_ + if `Flask `_ is not a framework of choice. + + +Use cases +--------- + +Use cases that need to be implemented by webhook2fedmsg. + + +* User login + + User will be able to login through web interface using FAS account. + +* User can add token for service + + When logged in user can add token for any service known by webhook2fedmsg. We should support multiple tokens per service. + +* User can modify/remove existing tokens + + When logged in user should be able to modify/remove tokens for services he added earlier. + +* User can list already added tokens + + When logged in user should be able to see tokens he already added and for which service they are. + +* User can see the URL for webhook + + webhook2fedmsg should show on web interface URL that could be used to set up webhook on the service. + +* Listening for webhook POST requests + + The application will provide endpoint for incoming requests. Ideally one per service. + The webhook needs to be set by user on webhook service page. + +* Validating webhook POST requests + + Every incoming request from webhook services will be validated against the token database and search if it is valid token for service. + It will drop any invalid request. + +* Processing webhook POST requests + + Each message needs to be converted from webhook JSON to Fedora message. The webhook2fedmsg should provide Fedora messaging schema for topics emitted by it. + + +Endpoints +--------- + +Endpoints that should be provided by webhook2fedmsg. + +* `/` + + Index page. User is able to login here. + +* `/tokens` + + Token management page shows most information relevant to user - URLs for webhook endpoints, management of tokens. + +* `/login` + + Login page user is automatically redirected to, if not authenticated. + +* `/` + + Endpoint where webhook service should sent POST requests. + +* `/api` + + Endpoint for API requests. API will be a nice to have alternative to web GUI. diff --git a/docs/webhook2fedmsg/index.rst b/docs/webhook2fedmsg/index.rst new file mode 100644 index 0000000..180376e --- /dev/null +++ b/docs/webhook2fedmsg/index.rst @@ -0,0 +1,100 @@ +webhook2fedmsg +============= + +Purpose +------- + +This investigation's goal is to explore a possible new application that will +convert webhooks from different services to Fedora messaging messages. + +Resources +--------- + +* Initiative proposal: https://pagure.io/cpe/initiatives-proposal/issue/18 + +Requirements +------------ + +* User can log in using FAS account +* User can link different webhook services to FAS account +* Application will listen for the webhook POST requests +* Application will provide endpoint for each service +* POST request will be validated against the known tokens for the service +* Support for `GitHub `_ +* Support for `Discourse `_ +* CI - Recommending `Zuul CI `_ + or `Github Actions `_ +* Documentation - Documentation could be hosted on `readthedocs.org `_ +* OpenShift deployment +* Unit tests - Recommending `tox `_ +* Fedora messaging schema +* Easy to use dev env - Recommending either `vagrant `_ + or containerized environment + + +Nice to have +------------ + +List of features that would be nice to have. + +* Document how to create webhooks on every service +* Document how to generate token on every service +* Support for `GitLab `_ +* Support for `MediaWiki `_ +* Automatic dependency management - Recommending `renovate `_ +* API for managing tokens +* Automatic deployment in OpenShift + + +Investigation +------------- + +Following it the design document for webhook2fedmsg. + +.. toctree:: + :maxdepth: 1 + + design_of_webhook2fedmsg + + +The Good Points +--------------- + +1. Supports multiple webhook services +2. Easier to maintain - less codebases +3. Easier to add new webhook service in future +4. Users can manage tokens by themselves +5. With this solution we don't need to rewrite github2fedmsg - one less initiative + + +The Bad points +-------------- + +1. Initial work - completely new app +2. Another application to maintain - However this will allow us to get rid of few others +2. Manual work with tokens and webhooks on webhook service side for users + + +Conclusions +----------- + +This application will help us to lower our codebase. We wouldn't need to maintain multiple +applications service2fedmsg anymore, but everything that provides webhook could +be managed by this application. +It is definitely something we have knowledge and skills to do. + +Proposed Roadmap +---------------- + +* Step 1 - Write new app with tests and documentation +* Step 2 - Create separate repository with message schemas +* Step 3 - Deploy in staging +* Step 4 - Inform community this service is available for testing +* Step 5 - Deploy in production +* Step 6 - Community blog announcing this new service + +Estimate of work +---------------- + +This work will need at least 2 developers, one with web frontend skill set. +The estimation for this project is 12 weeks.