Add webhook2fedmsg investigation

This commit contains investigation for
https://pagure.io/cpe/initiatives-proposal/issue/18

It describes a solution for new app converting webhooks to fedora messaging
messages.

Originally this was done as part of github2fedmsg investigation, but it is a
completely different proposal.

Signed-off-by: Michal Konečný <mkonecny@redhat.com>

docs/github2fedmsg/index.rst

@@ -55,7 +55,6 @@ Here is the list of ideas/things we discussed/looked at:
 
     revitalization
     rewrite
-    webhook2fedmsg
     enhancement_of_repository
 
 

docs/github2fedmsg/webhook2fedmsg.rst

@@ -1,29 +0,0 @@
-.. _webhook2fedmsg:
-
-webhook2fedmsg
-==============
-
-This document will investigate the possibility to create a more general solution
-converting various webhooks to fedora messaging messages. This will completely
-change the github2fedmsg to modular application supporting multiple sources.
-
-Requirements
-------------
-
-
-New design
-----------
-
-
-The Good Points
----------------
-
-1. Could support multiple sources in future
-2. Easier to maintain
-3. Easier to customize
-
-The Bad points
---------------
-
-1. More work for us
-2. Existing code will be thrown away

docs/initiatives.rst

@@ -23,6 +23,7 @@ Completed review
     flask-oidc/index
     communishift/index
     kerneltest/index
+    webhook2fedmsg/index
 
 Implemented
 -----------

docs/webhook2fedmsg/design_of_webhook2fedmsg.rst

@@ -0,0 +1,97 @@
+.. _design_of_webhook2fedmsg:
+
+Design document for webhook2fedmsg
+==================================
+
+This document will investigate the possibility to create a more general solution
+converting various webhooks to fedora messaging messages.
+
+
+External services
+-----------------
+
+External services webhook2fedmsg will communicate with.
+
+* Database
+
+  Will be used to store data about FAS user and tokens they added. We can consider supporting multiple tokens for each service per user.
+  For database it would be best to use existing `PostreSQL <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/postgresql_server>`_
+  instance we already host in Fedora infra.
+
+* Webhook services
+
+  The application will listen for webhooks from supported services. For each service should be separate backend, so it's easier to
+  remove/add new services.
+
+* FAS
+
+  We need to authenticate user against Fedora Account System.
+  For authentication we can use `flask-oidc <https://github.com/fedora-infra/flask-oidc>`_ or `authlib <https://docs.authlib.org/en/latest/>`_
+  if `Flask <https://flask.palletsprojects.com/en/2.2.x/>`_ is not a framework of choice.
+
+
+Use cases
+---------
+
+Use cases that need to be implemented by webhook2fedmsg.
+
+
+* User login
+
+  User will be able to login through web interface using FAS account.
+
+* User can add token for service
+
+  When logged in user can add token for any service known by webhook2fedmsg. We should support multiple tokens per service.
+
+* User can modify/remove existing tokens
+
+  When logged in user should be able to modify/remove tokens for services he added earlier.
+
+* User can list already added tokens
+
+  When logged in user should be able to see tokens he already added and for which service they are.
+
+* User can see the URL for webhook
+
+  webhook2fedmsg should show on web interface URL that could be used to set up webhook on the service.
+
+* Listening for webhook POST requests
+
+  The application will provide endpoint for incoming requests. Ideally one per service.
+  The webhook needs to be set by user on webhook service page.
+
+* Validating webhook POST requests
+
+  Every incoming request from webhook services will be validated against the token database and search if it is valid token for service.
+  It will drop any invalid request.
+
+* Processing webhook POST requests
+
+  Each message needs to be converted from webhook JSON to Fedora message. The webhook2fedmsg should provide Fedora messaging schema for topics emitted by it.
+
+
+Endpoints
+---------
+
+Endpoints that should be provided by webhook2fedmsg.
+
+* `/`
+
+  Index page. User is able to login here.
+
+* `/tokens`
+
+  Token management page shows most information relevant to user - URLs for webhook endpoints, management of tokens.
+
+* `/login`
+
+  Login page user is automatically redirected to, if not authenticated.
+
+* `/<service>`
+
+  Endpoint where webhook service should sent POST requests.
+
+* `/api`
+
+  Endpoint for API requests. API will be a nice to have alternative to web GUI.

docs/webhook2fedmsg/index.rst

@@ -0,0 +1,100 @@
+webhook2fedmsg
+=============
+
+Purpose
+-------
+
+This investigation's goal is to explore a possible new application that will
+convert webhooks from different services to Fedora messaging messages.
+
+Resources
+---------
+
+* Initiative proposal: https://pagure.io/cpe/initiatives-proposal/issue/18
+
+Requirements
+------------
+
+* User can log in using FAS account
+* User can link different webhook services to FAS account
+* Application will listen for the webhook POST requests
+* Application will provide endpoint for each service
+* POST request will be validated against the known tokens for the service
+* Support for `GitHub <https://docs.github.com/en/developers/webhooks-and-events/webhooks/creating-webhooks>`_
+* Support for `Discourse <https://meta.discourse.org/t/configure-webhooks-that-trigger-on-discourse-events-to-integrate-with-external-services/49045>`_
+* CI - Recommending `Zuul CI <https://fedoraproject.org/wiki/Zuul-based-ci>`_
+  or `Github Actions <https://github.com/features/actions>`_
+* Documentation - Documentation could be hosted on `readthedocs.org <https://readthedocs.org/>`_
+* OpenShift deployment
+* Unit tests - Recommending `tox <https://tox.wiki/en/latest/>`_
+* Fedora messaging schema
+* Easy to use dev env - Recommending either `vagrant <https://www.vagrantup.com/>`_
+  or containerized environment
+
+
+Nice to have
+------------
+
+List of features that would be nice to have.
+
+* Document how to create webhooks on every service
+* Document how to generate token on every service
+* Support for `GitLab <http://mmb.irbbarcelona.org/gitlab/help/web_hooks/web_hooks.md>`_
+* Support for `MediaWiki <https://www.mediawiki.org/wiki/Extension:EventBus>`_
+* Automatic dependency management - Recommending `renovate <https://docs.renovatebot.com/>`_
+* API for managing tokens
+* Automatic deployment in OpenShift
+
+
+Investigation
+-------------
+
+Following it the design document for webhook2fedmsg.
+
+.. toctree::
+    :maxdepth: 1
+
+    design_of_webhook2fedmsg
+
+
+The Good Points
+---------------
+
+1. Supports multiple webhook services
+2. Easier to maintain - less codebases
+3. Easier to add new webhook service in future
+4. Users can manage tokens by themselves
+5. With this solution we don't need to rewrite github2fedmsg - one less initiative
+
+
+The Bad points
+--------------
+
+1. Initial work - completely new app
+2. Another application to maintain - However this will allow us to get rid of few others
+2. Manual work with tokens and webhooks on webhook service side for users
+
+
+Conclusions
+-----------
+
+This application will help us to lower our codebase. We wouldn't need to maintain multiple
+applications service2fedmsg anymore, but everything that provides webhook could
+be managed by this application.
+It is definitely something we have knowledge and skills to do.
+
+Proposed Roadmap
+----------------
+
+* Step 1 - Write new app with tests and documentation
+* Step 2 - Create separate repository with message schemas
+* Step 3 - Deploy in staging
+* Step 4 - Inform community this service is available for testing
+* Step 5 - Deploy in production
+* Step 6 - Community blog announcing this new service
+
+Estimate of work
+----------------
+
+This work will need at least 2 developers, one with web frontend skill set.
+The estimation for this project is 12 weeks.