Infrastructure/arc
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

fix parsing errors and sphinx warnings

Browse source 
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
This commit is contained in:
Ryan Lercho 2023-11-16 08:02:56 +10:00 • committed by zlopez
parent 8fb9b2fdf0
commit ba720c3d77
98 changed files with 4799 additions and 4788 deletions
Download patch file Download diff file Expand all files Collapse all files

21
docs/arc_names.rst
View file

@ -2,15 +2,14 @@ ARC = f(days)
=============
As the title says ARC is an acronym that can vary through the days. Here is a
non-exhaustive list of all the significations we have found for ARC, we will
complete it as we find more.
non-exhaustive list of all the significations we have found for ARC, we will complete it
as we find more.
* Angry Rabbits Committee
* Advanced Repossession Crew
* Arbitrary Rejection Cabal
* ARC Research Committee
* Ancient Roman Chicken
* Alien Research Center
* Aoife's Raging Children
* Advanced Regional Circus
- Angry Rabbits Committee
- Advanced Repossession Crew
- Arbitrary Rejection Cabal
- ARC Research Committee
- Ancient Roman Chicken
- Alien Research Center
- Aoife's Raging Children
- Advanced Regional Circus

6
docs/badges/current_implementation.rst
View file

@ -1,10 +1,10 @@
.. _current_implementation:
Current implementation
====
======================
The Fedora Badges service is currently split up into multiple projects that all
work together to award badges to Fedora Project contributors. These parts are -
The Fedora Badges service is currently split up into multiple projects that all work
together to award badges to Fedora Project contributors. These parts are -
.. toctree::
:maxdepth: 1

16
docs/badges/current_implementation_artwork_and_rules.rst
View file

@ -1,14 +1,14 @@
.. _current_implementation_artwork_and_rules:
Artwork and rules
====
=================
The fedora-badges repo on Pagure is where community members contribute the
badge rules and badge images for new badges. These assets are added to the
fedora-badges repo, and copied over to the servers using the
`push-badges manual playbook <https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/manual/push-badges.yml>`_
The fedora-badges repo on Pagure is where community members contribute the badge rules
and badge images for new badges. These assets are added to the fedora-badges repo, and
copied over to the servers using the `push-badges manual playbook
<https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/manual/push-badges.yml>`_
The artwork is copied over to `badges-web01` and the rules are copied over to `badges-backend01`.
* Source repository: `https://pagure.io/fedora-badges/ <https://pagure.io/fedora-badges/>`_
The artwork is copied over to `badges-web01` and the rules are copied over to
`badges-backend01`.
- Source repository: https://pagure.io/fedora-badges/

24
docs/badges/current_implementation_badge_issuer_scripts.rst
View file

@ -1,19 +1,19 @@
.. _current_implementation_badge_issuer_scripts:
Badge issuer scripts
====
====================
In addition to manually granting a badge in the web interface, and the
messages consumer granting badges, the third way badges are granted are by a
collection of scripts that live on the badges-backend01 server.
In addition to manually granting a badge in the web interface, and the messages consumer
granting badges, the third way badges are granted are by a collection of scripts that
live on the badges-backend01 server.
These scripts run periodically using cron, and are used for badges that aren't
triggered by an incoming message on the messaging bus.
These scripts run periodically using cron, and are used for badges that aren't triggered
by an incoming message on the messaging bus.
A good example of this is the "lifecycle" badges, that award people for having
a Fedora Account for a period of time. There is no message sent when a Fedora
Account turns a certain age, so this is done using a script.
* Source repository: `https://pagure.io/fedora-infra/ansible/blob/main/f/roles/badges/backend/files/cron <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/badges/backend/files/cron>`_
* Production location: badges-backend01.iad2.fedoraproject.org
A good example of this is the "lifecycle" badges, that award people for having a Fedora
Account for a period of time. There is no message sent when a Fedora Account turns a
certain age, so this is done using a script.
- Source repository:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/badges/backend/files/cron
- Production location: badges-backend01.iad2.fedoraproject.org

58
docs/badges/current_implementation_databases_and_api.rst
View file

@ -1,37 +1,45 @@
.. _current_implementation_databases_and_api:
The database and the API
====
========================
The database for Fedora Badges stores the badge definitions. and what users
have been granted these badges. These is also a Python API for interacting
with the database.
The database for Fedora Badges stores the badge definitions. and what users have been
granted these badges. These is also a Python API for interacting with the database.
* Source repository: `https://github.com/fedora-infra/tahrir-api <https://github.com/fedora-infra/tahrir-api>`_
* Production location: db01.iad2.fedoraproject.org/tahrir
- Source repository: https://github.com/fedora-infra/tahrir-api
- Production location: db01.iad2.fedoraproject.org/tahrir
Technology used
----
---------------
The database and API is written in Python 2 and uses SQLAlchemy for database
management. In the production deployment, a Postgres database is used.
The database and API is written in Python 2 and uses SQLAlchemy for database management.
In the production deployment, a Postgres database is used.
Basic table information
----
-----------------------
The badges database appears to have been implemented with extra features for
Open Badges (or Badgr) compatibility, but a lot of these tables are not really
in use in the production deployment of Fedora Badges.
The badges database appears to have been implemented with extra features for Open Badges
(or Badgr) compatibility, but a lot of these tables are not really in use in the
production deployment of Fedora Badges.
.. csv-table::
:header: "DB Table", "Information"
:widths: 15, 30
"Person", "A table for a user. Contains basic information about the user"
"Badge", "A table of all the badge definitions. Note that the badge image is not stored in the DB, it is just a link to an image on the backend server. Also, the criteria for a badge is only a URL to criteria."
"Assertion", "A table that maps users to badges allowing them to arbitrarily grant them."
"Authorization", "A table that maps users to badges allowing them to arbitrarily grant them."
"Issuer", "A table of the issuers that issue badges. In Fedora badges, there is only one issuer -- Fedora, so not really used."
"Series", "A table to do with having a series of badges -- can't find much documentation about it, but on production badges, it is empty, so not in use."
"Milestones", "Another table to do with having a series of badges -- can't find much documentation about it, but on production badges, it is empty, so not in use."
"Team", "Yet another table to do with having a series of badges -- can't find much documentation about it, but on production badges, it only has one entry 'Infrastructure'."
============= ==========================================================================
DB Table Information
============= ==========================================================================
Person A table for a user. Contains basic information about the user
Badge A table of all the badge definitions. Note that the badge image is not
stored in the DB, it is just a link to an image on the backend server.
Also, the criteria for a badge is only a URL to criteria.
Assertion A table that maps users to badges allowing them to arbitrarily grant them.
Authorization A table that maps users to badges allowing them to arbitrarily grant them.
Issuer A table of the issuers that issue badges. In Fedora badges, there is only
one issuer -- Fedora, so not really used.
Series A table to do with having a series of badges -- can't find much
documentation about it, but on production badges, it is empty, so not in
use.
Milestones Another table to do with having a series of badges -- can't find much
documentation about it, but on production badges, it is empty, so not in
use.
Team Yet another table to do with having a series of badges -- can't find much
documentation about it, but on production badges, it only has one entry
'Infrastructure'.
============= ==========================================================================

56
docs/badges/current_implementation_frontend.rst
View file

@ -1,48 +1,46 @@
.. _current_implementation_frontend:
Badges frontend
====
===============
Tahrir is the name given to the web interface that allows service users to
view the badges they have been awarded, view those awarded to other community
members, and grant special badges. Additionally, Fedora Badges admins also use
the web interface to add new badges into the database.
Tahrir is the name given to the web interface that allows service users to view the
badges they have been awarded, view those awarded to other community members, and grant
special badges. Additionally, Fedora Badges admins also use the web interface to add new
badges into the database.
* Source repository: `https://github.com/fedora-infra/tahrir <https://github.com/fedora-infra/tahrir>`_
* Production deployment: `https://badges.fedoraproject.org/ <https://badges.fedoraproject.org/>`_
* Production location: `badges-web01.iad2.fedoraproject.org <badges-web01.iad2.fedoraproject.org>`_
- Source repository: https://github.com/fedora-infra/tahrir
- Production deployment: https://badges.fedoraproject.org/
- Production location: badges-web01.iad2.fedoraproject.org
Note: The
`badges artworks <https://pagure.io/fedora-badges/blob/master/f/pngs>`_ are
stored in the `Fedora Badges repository <https://pagure.io/Fedora-Badges>`_,
and when new rules are added, copied over to the production location (i.e.
badges-web01 server) using the
`push-badges manual playbook <https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/manual/push-badges.yml>`_.
Note: The `badges artworks <https://pagure.io/fedora-badges/blob/master/f/pngs>`_ are
stored in the `Fedora Badges repository <https://pagure.io/Fedora-Badges>`_, and when
new rules are added, copied over to the production location (i.e. badges-web01 server)
using the `push-badges manual playbook
<https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/manual/push-badges.yml>`_.
Technology used
----
---------------
The web interface is written in Python 2 and uses Pyramid as a web framework
with Mako templates.
The web interface is written in Python 2 and uses Pyramid as a web framework with Mako
templates.
Usecases
----
--------
The following are the usecases of the web interface of Fedora Badges.
User usecases
^^^^
~~~~~~~~~~~~~
* View the badges that a particular user has
* View the leaderboard of badges
* View the details of a specific badge
* Get granted a badge (via a link or QR code)
* Give a badge to another user (if I have permissions on that badge)
- View the badges that a particular user has
- View the leaderboard of badges
- View the details of a specific badge
- Get granted a badge (via a link or QR code)
- Give a badge to another user (if I have permissions on that badge)
Admin usecases
^^^^
* Add a new badge to the database
* Give grant permissions to a user for a specific badge
* Create an invitation so users can get a badge via a QR code / link
~~~~~~~~~~~~~~
- Add a new badge to the database
- Give grant permissions to a user for a specific badge
- Create an invitation so users can get a badge via a QR code / link

54
docs/badges/current_implementation_messages_consumer.rst
View file

@ -1,40 +1,38 @@
.. _current_implementation_messages_consumer:
Messages consumer
====
=================
The fedbadges fedmsg consumer is the main piece of Fedora Badges that grants
badges to the service users. It consumes every message from the fedmsg bus,
checks to see if that message matches a badge rule, and if it does, uses the
Tahrir API (database API) to issue the badge to the user in the database.
The fedbadges fedmsg consumer is the main piece of Fedora Badges that grants badges to
the service users. It consumes every message from the fedmsg bus, checks to see if that
message matches a badge rule, and if it does, uses the Tahrir API (database API) to
issue the badge to the user in the database.
Many badge rules require the consumer to know about past messages too, not
just the one that triggered the rule. For example, a badge for 100 Bodhi
comments requires a historical count of bodhi comment messages from the
Datanommer (every message sent, ever) database. Note, however that currently
the consumer interfaces directly with the Datanommer database, rather than
using Datagrepper (been assumed so for performance reasons).
Many badge rules require the consumer to know about past messages too, not just the one
that triggered the rule. For example, a badge for 100 Bodhi comments requires a
historical count of bodhi comment messages from the Datanommer (every message sent,
ever) database. Note, however that currently the consumer interfaces directly with the
Datanommer database, rather than using Datagrepper (been assumed so for performance
reasons).
The badges rules are defined in YAML files are defined in a custom
specification that is documented in the
`fedbadges documentation <https://github.com/fedora-infra/fedbadges/blob/develop/README.rst>`_.
For Fedora Badges, the
`badges rules are stored <https://pagure.io/fedora-badges/blob/master/f/rules>`_
in the `Fedora Badges repo <https://pagure.io/Fedora-Badges>`_ and when new
rules are added, copied over to the badges-backend01 server using the
`pushes-badges manual playbook <https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/manual/push-badges.yml>`_.
The badges rules are defined in YAML files are defined in a custom specification that is
documented in the `fedbadges documentation
<https://github.com/fedora-infra/fedbadges/blob/develop/README.rst>`_. For Fedora
Badges, the `badges rules are stored
<https://pagure.io/fedora-badges/blob/master/f/rules>`_ in the `Fedora Badges repo
<https://pagure.io/Fedora-Badges>`_ and when new rules are added, copied over to the
badges-backend01 server using the `pushes-badges manual playbook
<https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/manual/push-badges.yml>`_.
* Source repository: `https://github.com/fedora-infra/fedbadges <https://github.com/fedora-infra/fedbadges>`_
* Production location: badges-backend01.iad2.fedoraproject.org
- Source repository: https://github.com/fedora-infra/fedbadges
- Production location: badges-backend01.iad2.fedoraproject.org
Note: The messages consumer has some code in the main development branch that
appears to start implementing support for Badgr. but this appears to be
half-finished and was never deployed to production. It also appears to only
send badges to badgr, and not store them in the tahrir database. So content
can start here.
Note: The messages consumer has some code in the main development branch that appears to
start implementing support for Badgr. but this appears to be half-finished and was never
deployed to production. It also appears to only send badges to badgr, and not store them
in the tahrir database. So content can start here.
Technologies used
----
-----------------
The messages consumer is written in Python 2.

199
docs/badges/discourse_integration.rst
View file

@ -1,122 +1,123 @@
.. _discourse_integration:
Integrate Fedora Discussions with the Fedora Badges System
====
==========================================================
Fedora Badges has not only been a means to incentivize contribution for the
Fedora Project community but also been an topic of active conversation and a
healthy competition among the community members. One of the attempts to bring
the system to the forefront was made by
`Matthew Miller <https://accounts.fedoraproject.org/user/mattdm>`_ on
`this Fedora Discussions thread <https://discussion.fedoraproject.org/t/magical-experimental-fedora-badges-topic/35007>`_
and participated by a lot of community members. This turned out to be quite a
successful experiment and the badges that people have acquired got integrated
with the Discourse badges page (For eg. here is a catalog of badges acquired by
`Akashdeep Dhar <https://discussion.fedoraproject.org/u/t0xic0der/badges>`_).
Fedora Badges has not only been a means to incentivize contribution for the Fedora
Project community but also been an topic of active conversation and a healthy
competition among the community members. One of the attempts to bring the system to the
forefront was made by `Matthew Miller <https://accounts.fedoraproject.org/user/mattdm>`_
on `this Fedora Discussions thread
<https://discussion.fedoraproject.org/t/magical-experimental-fedora-badges-topic/35007>`_
and participated by a lot of community members. This turned out to be quite a successful
experiment and the badges that people have acquired got integrated with the Discourse
badges page (For eg. here is a catalog of badges acquired by `Akashdeep Dhar
<https://discussion.fedoraproject.org/u/t0xic0der/badges>`_).
As it stands, the usage of Discourse (or Fedora Discussions) happens to help
with simplifying the creation (and maintenance) of the frontend of the Fedora
Badges system. Although this is a welcome proposal, the frontend happens to be
the least challenging component of the system (as mentioned by Ryan Lerch in
`his reply <https://discussion.fedoraproject.org/t/discourse-badges-should-we-or-shouldnt-we/34698/6>`_
to the
`original post <https://discussion.fedoraproject.org/t/discourse-badges-should-we-or-shouldnt-we/34698>`_
about testing waters around
`Discourse Badges <http://discussion.fedoraproject.org/badges>`_ created on
23rd Nov 2021) but the fact that it still has not been maintained and requires
modernization cannot be denied. At the very same time, it is a very alluring
choice which helps us save efforts in developing the frontend from ground up
and redirect them into redeveloping the backend, which admittedly requires the
most attention.
As it stands, the usage of Discourse (or Fedora Discussions) happens to help with
simplifying the creation (and maintenance) of the frontend of the Fedora Badges system.
Although this is a welcome proposal, the frontend happens to be the least challenging
component of the system (as mentioned by Ryan Lerch in `his reply
<https://discussion.fedoraproject.org/t/discourse-badges-should-we-or-shouldnt-we/34698/6>`_
to the `original post
<https://discussion.fedoraproject.org/t/discourse-badges-should-we-or-shouldnt-we/34698>`_
about testing waters around `Discourse Badges
<http://discussion.fedoraproject.org/badges>`_ created on 23rd Nov 2021) but the fact
that it still has not been maintained and requires modernization cannot be denied. At
the very same time, it is a very alluring choice which helps us save efforts in
developing the frontend from ground up and redirect them into redeveloping the backend,
which admittedly requires the most attention.
The original post about exploring the idea of using Discourse as the frontend
for Fedora Badges system (which can be found
`here <https://discussion.fedoraproject.org/t/exploring-the-idea-of-using-discourse-as-a-badges-frontend-the-backend-implications/34887>`_
created on 1st Dec 2021) further details on the proposed approaches of making
it happen. Although the study is superficial but the implications provided
there should be adequate enough to connect with our study regarding the
entities involved in the proposed redevelopment of the system (the one that can
be found
`here <https://fedora-arc.readthedocs.io/en/latest/badges/prop_rewrite_entities.html>`_
) to deduce that Fedora Discussions could very well be used to be a drop-in
replacement of sorts for our Liberation entity (information about the internal
entities can be found
`here <https://fedora-arc.readthedocs.io/en/latest/badges/prop_rewrite_entities.html#internal-entities>`_
The original post about exploring the idea of using Discourse as the frontend for Fedora
Badges system (which can be found `here
<https://discussion.fedoraproject.org/t/exploring-the-idea-of-using-discourse-as-a-badges-frontend-the-backend-implications/34887>`__
created on 1st Dec 2021) further details on the proposed approaches of making it happen.
Although the study is superficial but the implications provided there should be adequate
enough to connect with our study regarding the entities involved in the proposed
redevelopment of the system (the one that can be found `here
<https://fedora-arc.readthedocs.io/en/latest/badges/prop_rewrite_entities.html>`__ ) to
deduce that Fedora Discussions could very well be used to be a drop-in replacement of
sorts for our Liberation entity (information about the internal entities can be found
`here
<https://fedora-arc.readthedocs.io/en/latest/badges/prop_rewrite_entities.html#internal-entities>`__
).
The discussions in the aforementioned post were continued over at the original
post sharing notes of the things needed to be taken care of while implementing
the same (which can be found
`here <https://discussion.fedoraproject.org/t/badges-to-discourse-experiment-notes-on-discourse-as-a-badges-front-end/35262>`_
created on 16th Dec 2021). The following are the key points we derived and
discussed about from the said treatise.
The discussions in the aforementioned post were continued over at the original post
sharing notes of the things needed to be taken care of while implementing the same
(which can be found `here
<https://discussion.fedoraproject.org/t/badges-to-discourse-experiment-notes-on-discourse-as-a-badges-front-end/35262>`_
created on 16th Dec 2021). The following are the key points we derived and discussed
about from the said treatise.
1. The Discourse frontend does provide for a much faster loading of badges list
(Source:
`JSON acquring time for user mattdm <https://discussion.fedoraproject.org/t/badges-to-discourse-experiment-notes-on-discourse-as-a-badges-front-end/35262#:~:text=Getting%20list%20of%20badges%20for%20mattdm%20via%20JSON%20here%3A%20about%20140ms.%20From%20badges%20site%2C%20about%2035%20whole%20seconds>`_
) and the plugin system is convenient enough to allow for adding features
and elements atop a stock interface.
2. The global listing of badges on the Discourse interface allows marking of
acquired badges and for grouping, which can help solve the issue of
introducing badge paths, although the listing of badges according to their
age needs looking into.
3. The per-user listing of badges could use some organization according to
categories and/or badge paths but favourite badges can be picked and shown
on the profile card, which makes these badges more than a royal collectible
but something to start discussions around.
4. The leaderboards also remain something to be seen, though the implementation
should be fairly easy if we are to have the same format as that of the
current frontend (i.e. show weekly, monthly and all-time leaders). The badge
count for the users should be added too.
1. The Discourse frontend does provide for a much faster loading of badges list (Source:
`JSON acquring time for user mattdm
<https://discussion.fedoraproject.org/t/badges-to-discourse-experiment-notes-on-discourse-as-a-badges-front-end/35262#:~:text=Getting%20list%20of%20badges%20for%20mattdm%20via%20JSON%20here%3A%20about%20140ms.%20From%20badges%20site%2C%20about%2035%20whole%20seconds>`_
) and the plugin system is convenient enough to allow for adding features and
elements atop a stock interface.
2. The global listing of badges on the Discourse interface allows marking of acquired
badges and for grouping, which can help solve the issue of introducing badge paths,
although the listing of badges according to their age needs looking into.
3. The per-user listing of badges could use some organization according to categories
and/or badge paths but favourite badges can be picked and shown on the profile card,
which makes these badges more than a royal collectible but something to start
discussions around.
4. The leaderboards also remain something to be seen, though the implementation should
be fairly easy if we are to have the same format as that of the current frontend
(i.e. show weekly, monthly and all-time leaders). The badge count for the users
should be added too.
5. The use of Discourse's internal badge levels alongside our badge paths has a
possibility of doing more harm than good, as it allegedly allows for only
three levels (i.e. bronze, silver and gold). This might confuse the frontend
if certain badges of a path belong to a level or tier.
6. The UI for manually awarding badges is allegedly restricted to only those
who are admins of the forum. Integrating that access capacity with a FAS
group should allow for members belonging to that group to award badges while
restricting them from any other admin-like operations.
7. The inclusion of QR codes or links to receive badges does not seem to be a
feature on the Discourse frontend but it is something that we could play
around with and implement using automation scripts. Also, there should be a
way for folks to opt-in or opt-out of these as they fit.
possibility of doing more harm than good, as it allegedly allows for only three
levels (i.e. bronze, silver and gold). This might confuse the frontend if certain
badges of a path belong to a level or tier.
6. The UI for manually awarding badges is allegedly restricted to only those who are
admins of the forum. Integrating that access capacity with a FAS group should allow
for members belonging to that group to award badges while restricting them from any
other admin-like operations.
7. The inclusion of QR codes or links to receive badges does not seem to be a feature on
the Discourse frontend but it is something that we could play around with and
implement using automation scripts. Also, there should be a way for folks to opt-in
or opt-out of these as they fit.
The following are the pain-points we derived out of the study.
1. There are possibilities that the users would get bombarded with loads of
notifications as they receive loads of badges in a short duration of time.
If this is understood correctly, this issue can be safely ignored as this
happens to be a one-time problem only.
2. There can be times when the user can have their account activated for
Fedora Badges and not on Discourse, or vice-versa. We would require to
rework the database (and the access to it from the Discourse frontend) to
ensure that the Badges database has the precedence.
1. There are possibilities that the users would get bombarded with loads of
notifications as they receive loads of badges in a short duration of time. If this is
understood correctly, this issue can be safely ignored as this happens to be a
one-time problem only.
2. There can be times when the user can have their account activated for Fedora Badges
and not on Discourse, or vice-versa. We would require to rework the database (and the
access to it from the Discourse frontend) to ensure that the Badges database has the
precedence.
How to go about it?
----
-------------------
We swap out the Liberation entity part from the
`internal entity chart <https://fedora-arc.readthedocs.io/en/latest/_images/badges-proposed-architecture.png>`_
and replace it with the Discourse frontend. The interactions, for the most
parts, should stay the same but a considerably different set of technologies
would be required to work on the Discourse frontend.
We swap out the Liberation entity part from the `internal entity chart
<https://fedora-arc.readthedocs.io/en/latest/_images/badges-proposed-architecture.png>`_
and replace it with the Discourse frontend. The interactions, for the most parts, should
stay the same but a considerably different set of technologies would be required to work
on the Discourse frontend.
Is there a Plan B?
----
------------------
Of course, there is. Just in case, the proposal for the use of the Discourse
frontend is not met with the community's approval or if it becomes too
difficult to work on - we can always make the current implementation of
integration work with the redeveloped system. In this way, we can have our own
frontend all while ensuring that our badges are not left behind catching
electronic dust by giving them the attention on the Fedora Discussions forum,
with their use as custom titles and listing them on the user cards.
Of course, there is. Just in case, the proposal for the use of the Discourse frontend is
not met with the community's approval or if it becomes too difficult to work on - we can
always make the current implementation of integration work with the redeveloped system.
In this way, we can have our own frontend all while ensuring that our badges are not
left behind catching electronic dust by giving them the attention on the Fedora
Discussions forum, with their use as custom titles and listing them on the user cards.
Further reading
----
---------------
1. `Discourse Badges - should we or shouldn't we? <https://discussion.fedoraproject.org/t/discourse-badges-should-we-or-shouldnt-we/34698>`_
1. `Discourse Badges - should we or shouldn't we?
<https://discussion.fedoraproject.org/t/discourse-badges-should-we-or-shouldnt-we/34698>`_
2. `Badges on Fedora Discussions <https://discussion.fedoraproject.org/badges>`_
3. `Exploring the idea of using Discourse as a badges frontend — the backend implications <https://discussion.fedoraproject.org/t/exploring-the-idea-of-using-discourse-as-a-badges-frontend-the-backend-implications/34887>`_
4. `Magical Experimental Fedora Badges Topic! <https://discussion.fedoraproject.org/t/magical-experimental-fedora-badges-topic/35007>`_
5. `Badges-to-Discourse experiment... notes on Discourse as a badges front-end <https://discussion.fedoraproject.org/t/badges-to-discourse-experiment-notes-on-discourse-as-a-badges-front-end/35262>`_
3. `Exploring the idea of using Discourse as a badges frontend — the backend
implications
<https://discussion.fedoraproject.org/t/exploring-the-idea-of-using-discourse-as-a-badges-frontend-the-backend-implications/34887>`_
4. `Magical Experimental Fedora Badges Topic!
<https://discussion.fedoraproject.org/t/magical-experimental-fedora-badges-topic/35007>`_
5. `Badges-to-Discourse experiment... notes on Discourse as a badges front-end
<https://discussion.fedoraproject.org/t/badges-to-discourse-experiment-notes-on-discourse-as-a-badges-front-end/35262>`_

123
docs/badges/exploring_the_development_environment.rst
View file

@ -1,104 +1,101 @@
.. _exploring_the_development_environment:
Exploring the development environment
====
=====================================
This document will go through all repositories of Badges project
and describe the state of development environment and how it could be improved.
This document will go through all repositories of Badges project and describe the state
of development environment and how it could be improved.
Tahrir
----
------
`Tahrir <https://github.com/fedora-infra/tahrir>`_ is a Fedora badges frontend.
The guide for setting up development environment is described in
`DEVELOPING.md <https://github.com/fedora-infra/tahrir/blob/develop/DEVELOPING.md>`_
and it's using `Vagrant <https://www.vagrantup.com/>`_.
`Tahrir <https://github.com/fedora-infra/tahrir>`_ is a Fedora badges frontend. The
guide for setting up development environment is described in `DEVELOPING.md
<https://github.com/fedora-infra/tahrir/blob/develop/DEVELOPING.md>`_ and it's using
`Vagrant <https://www.vagrantup.com/>`_.
Vagrantfile.example is using F29, which is no longer available. Updated this to F36,
but encountered issue when tried to provision the machine by ansible
`Failed to connect to the host via ssh: Shared connection to xxx.xxx.xxx.xxx closed.`
Solved by defining the domain of VM in Vagrantfile::
Vagrantfile.example is using F29, which is no longer available. Updated this to F36, but
encountered issue when tried to provision the machine by ansible `Failed to connect to
the host via ssh: Shared connection to xxx.xxx.xxx.xxx closed.` Solved by defining the
domain of VM in Vagrantfile:
config.vm.define "tahrir" do |tahrir|
tahrir.vm.host_name = "tahrir.example.com"
.. code-block::
tahrir.vm.provider :libvirt do |domain|
# Season to taste
domain.cpus = 4
domain.graphics_type = "spice"
domain.memory = 2048
domain.video_type = "qxl"
config.vm.define "tahrir" do |tahrir|
tahrir.vm.host_name = "tahrir.example.com"
tahrir.vm.provider :libvirt do |domain|
# Season to taste
domain.cpus = 4
domain.graphics_type = "spice"
domain.memory = 2048
domain.video_type = "qxl"
end
end
end
The installation of the tahrir needed a specific version of setuptools (57.5.0),
later version doesn’t have 2to3 in it.
The installation of the tahrir needed a specific version of setuptools (57.5.0), later
version doesn’t have 2to3 in it.
The provisioning steps in the ansible needed to be done manually, because of this setuptools issue.
Installation in a virtual environment inside the vagrant machine was finished successfully.
The provisioning steps in the ansible needed to be done manually, because of this
setuptools issue. Installation in a virtual environment inside the vagrant machine was
finished successfully.
Initialization of the database failed on
`ImportError: cannot import name 'ZopeTransactionExtension' from 'zope.sqlalchemy'`.
To get over this I needed to remove `extension=ZopeTransactionExtension()` and corresponding import
from `.venv/lib64/python3.8/site-packages/tahrir_api-0.8.1-py3.8.egg/tahrir_api/model.py`.
Initialization of the database failed on `ImportError: cannot import name
'ZopeTransactionExtension' from 'zope.sqlalchemy'`. To get over this I needed to remove
`extension=ZopeTransactionExtension()` and corresponding import from
`.venv/lib64/python3.8/site-packages/tahrir_api-0.8.1-py3.8.egg/tahrir_api/model.py`.
Next issue during the initialization of DB was `ModuleNotFoundError: No module named 'utils'`
Fixed this by adding `.` before local imports in
Next issue during the initialization of DB was `ModuleNotFoundError: No module named
'utils'` Fixed this by adding `.` before local imports in
`.venv/lib64/python3.8/site-packages/tahrir_api-0.8.1-py3.8.egg/tahrir_api/dbapi.py`
This was the end of the provisioning phase.
When trying to run `pserve --reload development.ini` I ended up with error
`TypeError: SignedCookieSessionFactory() got an unexpected keyword argument 'cookie_secure'`
Fixed by removing `cookie_` prefix from
`SignedCookieSessionFactory` call in `/vagrant/tahrir/__init__.py`
When trying to run `pserve --reload development.ini` I ended up with error `TypeError:
SignedCookieSessionFactory() got an unexpected keyword argument 'cookie_secure'` Fixed
by removing `cookie_` prefix from `SignedCookieSessionFactory` call in
`/vagrant/tahrir/__init__.py`
Another error
`ImportError: cannot import name 'authenticated_userid' from 'pyramid.security' (unknown location)`
Fixed this by changing the dependency `pyramid < 2.0` in `setup.py`
See https://github.com/stevearc/pypicloud/issues/274 for more info
Another error `ImportError: cannot import name 'authenticated_userid' from
'pyramid.security' (unknown location)` Fixed this by changing the dependency `pyramid <
2.0` in `setup.py` See https://github.com/stevearc/pypicloud/issues/274 for more info
After that the instance is running, but it’s not accessible outside of the VM.
When trying curl http://0.0.0.0:8000 it throws error
`TypeError: __init__() got an unexpected keyword argument 'extension'`
After that the instance is running, but it’s not accessible outside of the VM. When
trying curl http://0.0.0.0:8000 it throws error `TypeError: __init__() got an unexpected
keyword argument 'extension'`
At this point we wouldn't want to put more effort in it, because it means starting to fix
issues in app itself and this is not part of the investigation.
At this point we wouldn't want to put more effort in it, because it means starting to
fix issues in app itself and this is not part of the investigation.
Recommendation
^^^^
~~~~~~~~~~~~~~
Before fixing the development environment Tahrir needs to be updated to python 3.0, so
it could run no never versions of Fedora.
The development environment itself could be used with few enhancements. See investigation
above.
The development environment itself could be used with few enhancements. See
investigation above.
tahrir_api
----
----------
tahrir_api is part of Fedora Badges backend and provides API and database to work with.
The guide for setting up development environment is available in
`README <https://github.com/fedora-infra/tahrir-api#readme>`_ and it's using Python
virtual environment.
The guide for setting up development environment is available in `README
<https://github.com/fedora-infra/tahrir-api#readme>`_ and it's using Python virtual
environment.
The guide is working without issue, but it's missing any tools to work with tahrir_api
itself.
Recommendation
^^^^
Provide guide or tools to work with standalone tahrir_api in development environment.
**Recommendation**
Provide guide or tools to work with standalone tahrir_api in development
environment.
fedbadges
----
---------
fedbadges is part of Fedora Badges backend and provides consumer for fedmsg messages.
There is no guide or any description of development environment.
Recommendation
^^^^
Create a development environment that allows you to work with fedbadges and document
it.
**Recommendation**
Create a development environment that allows you to work with fedbadges and document
it.

121
docs/badges/index.rst
View file

@ -1,94 +1,93 @@
Fedora Badges
====
=============
Purpose
----
-------
Fedora Badges is a service that grants virtual accolades for milestones and
completing tasks within the Fedora Project community. For example, a community
member may collect badges for testing package updates on Bodhi when they test
1, 5, 10, 20, 40, 80, 125, 250, 500 and 1000 updates.
Fedora Badges is a service that grants virtual accolades for milestones and completing
tasks within the Fedora Project community. For example, a community member may collect
badges for testing package updates on Bodhi when they test 1, 5, 10, 20, 40, 80, 125,
250, 500 and 1000 updates.
Community members can also gather badges for attending events, or other badges
that are arbitrarily granted by users of the service, such as the badge the
Fedora Project Leader can grant whenever they want to.
Community members can also gather badges for attending events, or other badges that are
arbitrarily granted by users of the service, such as the badge the Fedora Project Leader
can grant whenever they want to.
Background
----
----------
It appears that Fedora Badges was initially designed closely in step with
`Open Badges <https://en.wikipedia.org/wiki/Mozilla_Open_Badges>`_ which now
has become `Badgr <https://badgr.com/>`_. As far as we can tell, the idea was
previously to be able to export Fedora Badges into Mozilla Backpack (or now,
Badgr Backpack) but this export facility was either never implemented or
deployed.
It appears that Fedora Badges was initially designed closely in step with `Open Badges
<https://en.wikipedia.org/wiki/Mozilla_Open_Badges>`_ which now has become `Badgr
<https://badgr.com/>`_. As far as we can tell, the idea was previously to be able to
export Fedora Badges into Mozilla Backpack (or now, Badgr Backpack) but this export
facility was either never implemented or deployed.
At some point in time, there also appears to have been an
`free and open source implementation of the Badgr server <https://github.com/concentricsky/badgr-server>`_
itself, which may have been a solution to replace the Fedora Badges setup, but
this project is either no longer available or set to be private on GitHub, so
it is not known what it was or is. For now, Badgr is just a service that
allows users to collate their badges from different issues into their
"backpack".
At some point in time, there also appears to have been an `free and open source
implementation of the Badgr server <https://github.com/concentricsky/badgr-server>`_
itself, which may have been a solution to replace the Fedora Badges setup, but this
project is either no longer available or set to be private on GitHub, so it is not known
what it was or is. For now, Badgr is just a service that allows users to collate their
badges from different issues into their "backpack".
Resources
----
---------
* `Fedora Badges backend <https://github.com/fedora-infra/fedbadges/>`_
* `Fedora Badges frontend <https://github.com/fedora-infra/tahrir>`_
* `Issue ticket about the ongoing efforts of revitalization <https://github.com/fedora-infra/fedbadges/issues/90>`_
* `Infrastructure developer guide <https://docs.fedoraproject.org/en-US/infra/developer_guide/>`_
* `Badges documentation in Fedora docs <https://docs.fedoraproject.org/en-US/badges/>`_
- `Fedora Badges backend <https://github.com/fedora-infra/fedbadges/>`_
- `Fedora Badges frontend <https://github.com/fedora-infra/tahrir>`_
- `Issue ticket about the ongoing efforts of revitalization
<https://github.com/fedora-infra/fedbadges/issues/90>`_
- `Infrastructure developer guide
<https://docs.fedoraproject.org/en-US/infra/developer_guide/>`_
- `Badges documentation in Fedora docs <https://docs.fedoraproject.org/en-US/badges/>`_
Index
----
-----
.. toctree::
:maxdepth: 1
current_implementation
exploring_the_development_environment
expectations_and_wishes
proposal_rewrite
proposal_revitalize
discourse_integration
Conclusions
----
-----------
Having explored both the options (i.e. of rewriting the project that consitute
the system from ground up and of revitalizing the existing codebase), we have
come to a conclusion that it makes a lot more sense to inherit and expand upon
the methodologies employed in the current codebase but implement the same from
groundup in a new functional rewrite.
Having explored both the options (i.e. of rewriting the project that consitute the
system from ground up and of revitalizing the existing codebase), we have come to a
conclusion that it makes a lot more sense to inherit and expand upon the methodologies
employed in the current codebase but implement the same from groundup in a new
functional rewrite.
Proposed roadmap
----
----------------
* Step 1 - Understand the project remit of the system to decide on the scope
* Step 2 - Get started with creating/inheriting the database schema for users
* Step 3 - Dump a snapshot of the existing database according to the new schema
* Step 4 - Brainstorm and document the interactions expected with the system
* Step 5 - Connect with the Fedora Design team to begin with the mockup designs
* Step 6 - Develop and document the Messages Consumer alongside Accolades API
* Step 7 - Test both against the dumped database and the existing Collection
* Step 8 - Develop and document the Accolades CLI and test its functionality
* Step 9 - Build Liberation frontend alongside the Fedora Websites & Apps Team
* Step 10 - Document and Test the frontend against the above mentioned entities
* Step 11 - Deploy and document the system on STAGING with the Fedora Infra team
* Step 12 - Release Accolades CLI as a Python package on PyPI and as RPM
* Step 13 - Announce public testing invites for the staging deployment
* Step 14 - Announce deprecation status for the existing Fedora Badges system
* Step 15 - Migrate database changes continually to keep up with the changes
* Step 16 - Switch the environment to production and... PROFIT?
- Step 1 - Understand the project remit of the system to decide on the scope
- Step 2 - Get started with creating/inheriting the database schema for users
- Step 3 - Dump a snapshot of the existing database according to the new schema
- Step 4 - Brainstorm and document the interactions expected with the system
- Step 5 - Connect with the Fedora Design team to begin with the mockup designs
- Step 6 - Develop and document the Messages Consumer alongside Accolades API
- Step 7 - Test both against the dumped database and the existing Collection
- Step 8 - Develop and document the Accolades CLI and test its functionality
- Step 9 - Build Liberation frontend alongside the Fedora Websites & Apps Team
- Step 10 - Document and Test the frontend against the above mentioned entities
- Step 11 - Deploy and document the system on STAGING with the Fedora Infra team
- Step 12 - Release Accolades CLI as a Python package on PyPI and as RPM
- Step 13 - Announce public testing invites for the staging deployment
- Step 14 - Announce deprecation status for the existing Fedora Badges system
- Step 15 - Migrate database changes continually to keep up with the changes
- Step 16 - Switch the environment to production and... PROFIT?
Of course, there are and will be more things to account for than what meets
the eye but from a Mount Everest high perspective - these are the steps we
would want to suggest in the roadmap.
Of course, there are and will be more things to account for than what meets the eye but
from a Mount Everest high perspective - these are the steps we would want to suggest in
the roadmap.
Estimate of work
----
----------------
Being a remake of a system with multiple internal entities working together,
it is estimated for the system to take at least 4 quarters to hit the staging
environment and 1 more quarter to make it to the production one.
Being a remake of a system with multiple internal entities working together, it is
estimated for the system to take at least 4 quarters to hit the staging environment and
1 more quarter to make it to the production one.

210
docs/badges/prop_rewrite_entities.rst
View file

@ -1,146 +1,132 @@
.. _prop_rewrite_entities:
Entities involved
====
=================
Let us dive deep into the entities that would come together to make the Fedora
Badges project functional. We will call them "entities" for the context of this
discussion. Also, the names used for the entities here are temporary in nature
and are, for the most parts, only representative of the functions that these
entities perform.
Let us dive deep into the entities that would come together to make the Fedora Badges
project functional. We will call them "entities" for the context of this discussion.
Also, the names used for the entities here are temporary in nature and are, for the most
parts, only representative of the functions that these entities perform.
Here is a diagram exhibiting the proposed architecture.
.. image:: ../_static/badges-proposed-architecture.png
:target: ../_images/badges-proposed-architecture.png
Internal entities
----
-----------------
**Accolades API** - Backend service
1. Succeeds the existing
`Tahrir <https://github.com/fedora-infra/tahrir>`_'s backend API and/or
is planned to be a reimplementation of the same.
2. This is planned to act as the backend service for the entire project and
has direct interactions with its neighbouring internal entities (i.e.
Collection, Database, Liberation, Messages Consumer, Accolades CLI).
1. Succeeds the existing `Tahrir <https://github.com/fedora-infra/tahrir>`_'s
backend API and/or is planned to be a reimplementation of the same.
2. This is planned to act as the backend service for the entire project and has
direct interactions with its neighbouring internal entities (i.e. Collection,
Database, Liberation, Messages Consumer, Accolades CLI).
**Accolades CLI** - Client application
1. Succeeds the existing
`Tahrir API <https://github.com/fedora-infra/tahrir-api>`_ client and/or
is planned to be a reimplementation of the same.
2. This is planned to act as a client application and/or library, usable for
automating purposes like pushing of badges (and their related awarding
conditions) to the Collection via the Accolades API, and other such high
level interactions offered by the Accolades API.
3. This is planned to be a standalone entity with just one possible
interaction with the neighbouring internal entity (i.e. Accolades API)
and three possible interactions with the neighbouring external entity
(i.e. community users, badge owners and service admins).
1. Succeeds the existing `Tahrir API <https://github.com/fedora-infra/tahrir-api>`_
client and/or is planned to be a reimplementation of the same.
2. This is planned to act as a client application and/or library, usable for
automating purposes like pushing of badges (and their related awarding
conditions) to the Collection via the Accolades API, and other such high level
interactions offered by the Accolades API.
3. This is planned to be a standalone entity with just one possible interaction with
the neighbouring internal entity (i.e. Accolades API) and three possible
interactions with the neighbouring external entity (i.e. community users, badge
owners and service admins).
**Liberation** - Web interface
1. Succeeds the existing
`Tahrir <https://github.com/fedora-infra/tahrir>`_'s frontend part and/or
is planned to be a reimplementation of the same.
2. This is planned to act as an interactive client application, accessible
via modern web browsers, usable by community members and service admins
for purposes like looking up the Fedora Badges leaderboard, checking
badges awarded to self or someone else etc. and adding new badges (and
their related awarding conditions) etc. respectively by interacting with
the Accolades API.
3. This is planned to be a standalone entity with just one possible
interaction with the neighbouring internal entity (i.e. Accolades API)
and three possible interactions with the neighbouring external entity
(i.e. community users, badge owners and service admins).
1. Succeeds the existing `Tahrir <https://github.com/fedora-infra/tahrir>`_'s
frontend part and/or is planned to be a reimplementation of the same.
2. This is planned to act as an interactive client application, accessible via
modern web browsers, usable by community members and service admins for purposes
like looking up the Fedora Badges leaderboard, checking badges awarded to self or
someone else etc. and adding new badges (and their related awarding conditions)
etc. respectively by interacting with the Accolades API.
3. This is planned to be a standalone entity with just one possible interaction with
the neighbouring internal entity (i.e. Accolades API) and three possible
interactions with the neighbouring external entity (i.e. community users, badge
owners and service admins).
**Collection** - Artworks and rules
1. Succeeds the existing `Fedora Badges <https://pagure.io/fedora-badges>`_
repository and/or is planned to be a reimplementation of the same.
2. This is planned to be a repository of supporting assets (artworks,
awarding conditions, consumer rules etc.) for the badges available for
being awarded to the community members, which can be read from and
updated by the interactions with the Accolades API entity and the
Liberation entity as well as by admins having relevant accesses to the
repository.
3. This is planned to be a standalone entity with just one possible
interaction with the neighbouring internal entity (i.e. Accolades API)
and one possible interaction with the neighbouring external entity (i.e.
repository members).
1. Succeeds the existing `Fedora Badges <https://pagure.io/fedora-badges>`_
repository and/or is planned to be a reimplementation of the same.
2. This is planned to be a repository of supporting assets (artworks, awarding
conditions, consumer rules etc.) for the badges available for being awarded to
the community members, which can be read from and updated by the interactions
with the Accolades API entity and the Liberation entity as well as by admins
having relevant accesses to the repository.
3. This is planned to be a standalone entity with just one possible interaction with
the neighbouring internal entity (i.e. Accolades API) and one possible
interaction with the neighbouring external entity (i.e. repository members).
**Database** - Achievements storage
1. Succeeds the existing Fedora Badges database storage for storing the
badges awarded to community members, date of felicitation and other
details and/or is planned to be a reimplementation of the same.
2. A huge chunk of the existing database is planned to be reimported into
the newer schema and the columns specific to the Open Badges (or Badgr)
compatibility will not be taken into consideration. This can be read
into and updated by the interactions with the Accolades API only.
3. This is planned to be a standalone entity with just one possible
interaction with the neighbouring internal entity (i.e. Accolades API)
and one possible interaction with the neighbouring external entity (i.e.
service admins).
1. Succeeds the existing Fedora Badges database storage for storing the badges
awarded to community members, date of felicitation and other details and/or is
planned to be a reimplementation of the same.
2. A huge chunk of the existing database is planned to be reimported into the newer
schema and the columns specific to the Open Badges (or Badgr) compatibility will
not be taken into consideration. This can be read into and updated by the
interactions with the Accolades API only.
3. This is planned to be a standalone entity with just one possible interaction with
the neighbouring internal entity (i.e. Accolades API) and one possible
interaction with the neighbouring external entity (i.e. service admins).
**Messages consumer** - Conditions listener
1. Succeeds the existing
`fedbadges <https://github.com/fedora-infra/fedbadges>`_ fedmsg listener
and/or is planned to be a reimplementation of the same.
2. This is planned to act as a messages consumer, listening into the Fedora
Messaging bus matching the schemas that it is configured for and
comparing the messages against the awarding conditions available on the
Collection, and on a successful match, making a request to the Accolades
API for awarding the said badge to a certain contributor who have met the
conditions specified in the related badge rule.
3. This is planned to be a standalone entity with two possible interactions
with the neighbouring internal entities (i.e. Accolades API and
Collection) and no possible interaction with the neighbouring external
entities.
1. Succeeds the existing `fedbadges <https://github.com/fedora-infra/fedbadges>`_
fedmsg listener and/or is planned to be a reimplementation of the same.
2. This is planned to act as a messages consumer, listening into the Fedora
Messaging bus matching the schemas that it is configured for and comparing the
messages against the awarding conditions available on the Collection, and on a
successful match, making a request to the Accolades API for awarding the said
badge to a certain contributor who have met the conditions specified in the
related badge rule.
3. This is planned to be a standalone entity with two possible interactions with the
neighbouring internal entities (i.e. Accolades API and Collection) and no
possible interaction with the neighbouring external entities.
External entities
----
-----------------
**Service admins** - Access Level IV
1. Consists of people belonging to the
`Badges Administrators <https://accounts.fedoraproject.org/group/sysadmin-badges/>`_
group.
2. People have access to the deployment environments (staging, development,
production etc.) and can control most of the aspects of all the internal
entities.
1. Consists of people belonging to the `Badges Administrators
<https://accounts.fedoraproject.org/group/sysadmin-badges/>`_ group.
2. People have access to the deployment environments (staging, development,
production etc.) and can control most of the aspects of all the internal
entities.
**Badge owners** - Access Level III
1. Consists of people who either lead or represent certain subcommunity, SIG
or subproject within the Fedora Project community, who want to make new
badges available and set rules for awarding.
2. People have an elevated access to the Liberation and Accolades CLI, which
enables them to access, create, change, award and delete badges that
they "own" (not those that they "have been awarded").
3. If there does not exist a designated IPA group for marking out these
entities, a new one can be created and they can be added to the same for
easing access control processes.
1. Consists of people who either lead or represent certain subcommunity, SIG or
subproject within the Fedora Project community, who want to make new badges
available and set rules for awarding.
2. People have an elevated access to the Liberation and Accolades CLI, which enables
them to access, create, change, award and delete badges that they "own" (not
those that they "have been awarded").
3. If there does not exist a designated IPA group for marking out these entities, a
new one can be created and they can be added to the same for easing access
control processes.
**Repository members** - Access Level II
1. Consists of people who either design, create and/or facilitate new badges
and want to make them available and set rules for awarding.
2. These are the people who can fall under the "Badge owners" category too
but that might not necessarily be the case always (for eg. badge
designers who do not lead or represent another subcommunity, SIG or
subproject but are tasked to create and upload badge assets).
3. There need not be a designated IPA group for marking out these entities
as their access is restricted to accessing, creating, changing and
deleting badges assets available on the Collection entity, which is
simply a version control system mediated repository.
1. Consists of people who either design, create and/or facilitate new badges and
want to make them available and set rules for awarding.
2. These are the people who can fall under the "Badge owners" category too but that
might not necessarily be the case always (for eg. badge designers who do not lead
or represent another subcommunity, SIG or subproject but are tasked to create and
upload badge assets).
3. There need not be a designated IPA group for marking out these entities as their
access is restricted to accessing, creating, changing and deleting badges assets
available on the Collection entity, which is simply a version control system
mediated repository.
**Community Users** - Access Level I
1. Consists of generally everyone from the Fedora Project community who have
a valid and activated, `IPA account <https://accounts.fedoraproject.org/>`_
and have signed the
`Fedora Project Contributor Agreement <https://accounts.fedoraproject.org/user/t0xic0der/settings/agreements/>`_.
2. These are the people who are eligible to only receive badges for their
recordable actions (i.e. those which either trigger a message on the
Fedora Messaging bus or are noticed by a Badge Owner entity) within the
Fedora Project community.
1. Consists of generally everyone from the Fedora Project community who have a valid
and activated, `IPA account <https://accounts.fedoraproject.org/>`_ and have
signed the `Fedora Project Contributor Agreement
<https://accounts.fedoraproject.org/user/t0xic0der/settings/agreements/>`_.
2. These are the people who are eligible to only receive badges for their recordable
actions (i.e. those which either trigger a message on the Fedora Messaging bus or
are noticed by a Badge Owner entity) within the Fedora Project community.
**NOTE** - These access levels are not exclusive in nature. A person who has an
Access Level IV, can (but not necessarily will) have those beneath it too and
by default, every FPCA signed member of the Fedora Project community, has at
least, Access Level I.
**NOTE** - These access levels are not exclusive in nature. A person who has an Access
Level IV, can (but not necessarily will) have those beneath it too and by default, every
FPCA signed member of the Fedora Project community, has at least, Access Level I.

319
docs/badges/prop_rewrite_interactions.rst
View file

@ -1,241 +1,228 @@
.. _prop_rewrite_interactions:
Interactions possible
====
=====================
The interactions between the previously mentioned entities would dictate the
workflow of the project and the direction of the flow of information among
multiple entities involved in an interaction. They are denoted by the line
connecting the entities and can be distinguished using the labels written on
the top of the aforementioned joining lines.
The interactions between the previously mentioned entities would dictate the workflow of
the project and the direction of the flow of information among multiple entities
involved in an interaction. They are denoted by the line connecting the entities and can
be distinguished using the labels written on the top of the aforementioned joining
lines.
Internal-only interactions
----
--------------------------
Interactions involving only internal entities are termed, in this context, as
internal-only interactions.
internal-only interactions.
.. image:: ../_static/badges-proposed-architecture.png
:target: ../_images/badges-proposed-architecture.png
AA
^^^^
~~
1. Defines the interaction between entities, Accolades CLI and Accolades API.
2. Accolades CLI can help abstract reading, creating, changing and deleting of
badge related assets or information with the help of the endpoints that the
Accolades API provides.
3. This interaction causes DA and CA interactions to take place, as the data
requested is read from and data modified is changed into the Database and
Collection entities. For eg. fetching information about the badges awarded
to a certain user from the Database entity (DA) and automating adding new
badges with the awarding conditions to the Collection entity (CA).
4. This interaction affects LA and MA interactions as changes made to the
Accolades API would affect how it reacts to the actions made by the
Liberation and Message Consumer entities. For eg. new badges added into the
Collection entity might show up on the catalog of the Liberation entity (LA)
and the awarding condition added to the Collection entity would change the
way the Message Consumer behaves in awarding badges (MA).
2. Accolades CLI can help abstract reading, creating, changing and deleting of badge
related assets or information with the help of the endpoints that the Accolades API
provides.
3. This interaction causes DA and CA interactions to take place, as the data requested
is read from and data modified is changed into the Database and Collection entities.
For eg. fetching information about the badges awarded to a certain user from the
Database entity (DA) and automating adding new badges with the awarding conditions to
the Collection entity (CA).
4. This interaction affects LA and MA interactions as changes made to the Accolades API
would affect how it reacts to the actions made by the Liberation and Message Consumer
entities. For eg. new badges added into the Collection entity might show up on the
catalog of the Liberation entity (LA) and the awarding condition added to the
Collection entity would change the way the Message Consumer behaves in awarding
badges (MA).
LA
^^^^
~~
1. Defines the interaction between entities, Liberation and Accolades API.
2. Liberation provides an interactive web interface for abstracting reading,
creating, changing, awarding and deleting of badge related assets or
information with the help of the endpoints that the Accolades API provides.
3. This interaction causes DA and CA interactions to take place, as the data
requested is read from and data modified is changed into the Database and
Collection entities. For eg. displaying the leaderboard of the top 100
community members by fetching it from the Database entity (DA) and adding
new badges with the awarding conditions to the Collection entity (CA).
4. This interaction affects AA and MA interactions as changes made to the
Accolades API would affect how it reacts to the actions made by the
Accolades CLI and Message Consumer entities. For eg. newly awarded badge
from Liberation would show up on the Accolades CLI for the user querying
about that information (AA) and the awarding condition added to the
Collection entity would change the way the Message Consumer behaves in
awarding badges (MA).
2. Liberation provides an interactive web interface for abstracting reading, creating,
changing, awarding and deleting of badge related assets or information with the help
of the endpoints that the Accolades API provides.
3. This interaction causes DA and CA interactions to take place, as the data requested
is read from and data modified is changed into the Database and Collection entities.
For eg. displaying the leaderboard of the top 100 community members by fetching it
from the Database entity (DA) and adding new badges with the awarding conditions to
the Collection entity (CA).
4. This interaction affects AA and MA interactions as changes made to the Accolades API
would affect how it reacts to the actions made by the Accolades CLI and Message
Consumer entities. For eg. newly awarded badge from Liberation would show up on the
Accolades CLI for the user querying about that information (AA) and the awarding
condition added to the Collection entity would change the way the Message Consumer
behaves in awarding badges (MA).
DA
^^^^
~~
1. Defines the interaction between entities, Database and Accolades API.
2. Database stores the information of the achievements made by the community
members by their recordable actions within the Fedora Project community,
along with the metadata of when the badges were awarded etc.
3. This interaction cannot be directly triggered and can only happen as a cause
of LA, AA and MA interactions. As a result, this interaction does not become
a causative agent for any other interaction. For eg. awarding a user with a
badge from the Liberation entity would lead to changes made in the Database
entity (LA), querying about the badges achieved by a user in the Accolades
CLI entity requires reading information from the Database entity (AA) and
conditioned awards made from the Message Consumer entity would be reflected
on the Database entity (MA).
4. This interaction affects only those interactions, that cause it to happen in
the first place. This can be explained as the consequences of the
aforementioned example interactions - for eg. Liberation entity would be
able to see the badge under the username that it was awarded to recently
(LA) and Accolades CLI entity would be able to retrieve the information it
requested for (AA) but the MA interaction would stay unaffected.
CA
^^^^
1. Defines the interaction between entities, Collection and Accolades API.
2. Collection entity stores the badge assets (artworks, badge rules, awarding
conditions etc.) that the Accolades API can read content from, add new
badges, change/update and delete/remove existing ones.
3. This interaction cannot be directly triggered and can only happen as a
cause of internal interactions like LA, AA, DA and MA as well as external
interactions like changes made into the Collection entity by external
entities like Repository Members etc. For eg. adding new badges for
distribution using the Liberation entity (LA) and doing so using the
Accolades CLI (AA) would lead to changes in the Collection entity, changes
in the database due to the awarding of badges would require reading
information from the Collection entity (DA) and rules on how the Message
Consumer entity would award badges would be described by the conditions
mentioned in the Collection entity (MA).
4. This interaction affects only those interactions, that cause it to happen
in the first place. This can be explained as the consequences of the
aforementioned examples - for eg. addition of new badge for distribution
would cause it to be accessible on Liberation entity (LA) and on Accolades
API (AA), Message Consumer entity would now have new rules as a result to
changes in the Collection entity (MA) but the DA interaction would stay
2. Database stores the information of the achievements made by the community members by
their recordable actions within the Fedora Project community, along with the metadata
of when the badges were awarded etc.
3. This interaction cannot be directly triggered and can only happen as a cause of LA,
AA and MA interactions. As a result, this interaction does not become a causative
agent for any other interaction. For eg. awarding a user with a badge from the
Liberation entity would lead to changes made in the Database entity (LA), querying
about the badges achieved by a user in the Accolades CLI entity requires reading
information from the Database entity (AA) and conditioned awards made from the
Message Consumer entity would be reflected on the Database entity (MA).
4. This interaction affects only those interactions, that cause it to happen in the
first place. This can be explained as the consequences of the aforementioned example
interactions - for eg. Liberation entity would be able to see the badge under the
username that it was awarded to recently (LA) and Accolades CLI entity would be able
to retrieve the information it requested for (AA) but the MA interaction would stay
unaffected.
MA
^^^^
CA
~~
1. Defines the interaction between entities, Messages Consumer and Accolades
API.
2. Message Consumer listens in to the Fedora Messaging bus and compares the
messages against the awarding conditions, and when the said condition is
satisfied, it makes a request to the Accolades API to award an external
entity, a certain Community User with the relevant badge.
3. This interaction cannot be directly triggered and can only happen as a
cause of internal interactions like CM and CA. For eg. any new additions
made to the Collection entity would change the way the Message Consumer
entity checks for conditions for badges (CA) and the Message Consumer would
read the content regarding the badges from the Collection entity (CM).
4. This interaction affects DA, LA and AA interactions as changes made to the
Accolades API entity by the Messages Consumer entity would affect how it
reacts to the actions made into the Database entity, by the Liberation
entity and Message Consumer entities. For eg. badges awarded automatically
by the Messsage Consumer entity's interaction would cause a change to be
made on the Database entity (DA) and lead them to be visible on the
Liberation entity (LA) and accessible on the Accolades CLI entity (AA).
1. Defines the interaction between entities, Collection and Accolades API.
2. Collection entity stores the badge assets (artworks, badge rules, awarding conditions
etc.) that the Accolades API can read content from, add new badges, change/update and
delete/remove existing ones.
3. This interaction cannot be directly triggered and can only happen as a cause of
internal interactions like LA, AA, DA and MA as well as external interactions like
changes made into the Collection entity by external entities like Repository Members
etc. For eg. adding new badges for distribution using the Liberation entity (LA) and
doing so using the Accolades CLI (AA) would lead to changes in the Collection entity,
changes in the database due to the awarding of badges would require reading
information from the Collection entity (DA) and rules on how the Message Consumer
entity would award badges would be described by the conditions mentioned in the
Collection entity (MA).
4. This interaction affects only those interactions, that cause it to happen in the
first place. This can be explained as the consequences of the aforementioned examples
- for eg. addition of new badge for distribution would cause it to be accessible on
Liberation entity (LA) and on Accolades API (AA), Message Consumer entity would now
have new rules as a result to changes in the Collection entity (MA) but the DA
interaction would stay unaffected.
MA
~~
1. Defines the interaction between entities, Messages Consumer and Accolades API.
2. Message Consumer listens in to the Fedora Messaging bus and compares the messages
against the awarding conditions, and when the said condition is satisfied, it makes a
request to the Accolades API to award an external entity, a certain Community User
with the relevant badge.
3. This interaction cannot be directly triggered and can only happen as a cause of
internal interactions like CM and CA. For eg. any new additions made to the
Collection entity would change the way the Message Consumer entity checks for
conditions for badges (CA) and the Message Consumer would read the content regarding
the badges from the Collection entity (CM).
4. This interaction affects DA, LA and AA interactions as changes made to the Accolades
API entity by the Messages Consumer entity would affect how it reacts to the actions
made into the Database entity, by the Liberation entity and Message Consumer
entities. For eg. badges awarded automatically by the Messsage Consumer entity's
interaction would cause a change to be made on the Database entity (DA) and lead them
to be visible on the Liberation entity (LA) and accessible on the Accolades CLI
entity (AA).
CM
^^^^
~~
1. Defines the interaction between entities, Collection and Message Consumer.
2. In order for the Message Consumer to obtain badge assets (artworks, rules,
awarding conditions etc.), it has to interact with the Collection entity.
3. This interaction cannot be directly triggered and can only happen as a cause
of internal interactions like CA and external interactions like Repository
Members making changes to the Collection entity. For eg. internal changes
made to the Collection entity and external changes made to the Collection
entity (for eg. by the Repository Members) would change the way the Messages
Consumer entity would check for awarding conditions for the badges (CA).
4. The interaction does not directly affect any other internal interaction but
can indirectly affect interactions like MA. For eg. changes in the
Collection entity can change the frequency and cases when the Messages
Consumer would interact with the Accolades API (MA).
2. In order for the Message Consumer to obtain badge assets (artworks, rules, awarding
conditions etc.), it has to interact with the Collection entity.
3. This interaction cannot be directly triggered and can only happen as a cause of
internal interactions like CA and external interactions like Repository Members
making changes to the Collection entity. For eg. internal changes made to the
Collection entity and external changes made to the Collection entity (for eg. by the
Repository Members) would change the way the Messages Consumer entity would check for
awarding conditions for the badges (CA).
4. The interaction does not directly affect any other internal interaction but can
indirectly affect interactions like MA. For eg. changes in the Collection entity can
change the frequency and cases when the Messages Consumer would interact with the
Accolades API (MA).
External-Internal interactions
----
------------------------------
Interactions involving both external and internal entities are termed, in this
context, as external-internal interactions.
Interactions involving both external and internal entities are termed, in this context,
as external-internal interactions.
Accolades API interacting with external entities
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. image:: ../_static/badges-proposed-api-ext-interactions.png
:target: ../_images/badges-proposed-api-ext-interactions.png
1. Service Admins [Access Level IV] would have a direct control on the
deployment environments (staging, development, production etc.).
1. Service Admins [Access Level IV] would have a direct control on the deployment
environments (staging, development, production etc.).
Accolades CLI interacting with external entities
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. image:: ../_static/badges-proposed-cli-ext-interactions.png
:target: ../_images/badges-proposed-cli-ext-interactions.png
1. Service Admins [Access Level IV] would have a direct control on the
deployment environments (staging, development, production etc.).
2. Badge Owners [Access Level III] would have an elevated access to the
Accolades API entity using the Accolades CLI entity where they can
automate addition of new badges for distribution within the community,
create, access, modify and remove the badges that they "own" (not the ones
that they "have been awarded").
3. Community Users [Access Level I] would have a limited access to the
Accolades API entity using the Accolades CLI entity where they can view
the badges received by them (and by others), access the catalog of
available badges, view leaderboards and other such similar tasks.
1. Service Admins [Access Level IV] would have a direct control on the deployment
environments (staging, development, production etc.).
2. Badge Owners [Access Level III] would have an elevated access to the Accolades API
entity using the Accolades CLI entity where they can automate addition of new badges
for distribution within the community, create, access, modify and remove the badges
that they "own" (not the ones that they "have been awarded").
3. Community Users [Access Level I] would have a limited access to the Accolades API
entity using the Accolades CLI entity where they can view the badges received by them
(and by others), access the catalog of available badges, view leaderboards and other
such similar tasks.
Liberation interacting with external entities
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. image:: ../_static/badges-proposed-web-ext-interactions.png
:target: ../_images/badges-proposed-web-ext-interactions.png
1. Service Admins [Access Level IV] would have a direct control on the
deployment environments (staging, development, production etc.).
2. Badge Owners [Access Level III] would have an elevated access to the
Accolades API entity using the Liberation entity where they can manually
add new badges for distribution within the community, create, access,
modify and remove the badges that they "own" (not the ones that they "have
been awarded").
3. Community Users [Access Level I] would have a limited access to the
Accolades API entity using the Liberation entity where they can view the
badges received by them (and by others), access the catalog of available
badges, view leaderboards and other such similar tasks.
1. Service Admins [Access Level IV] would have a direct control on the deployment
environments (staging, development, production etc.).
2. Badge Owners [Access Level III] would have an elevated access to the Accolades API
entity using the Liberation entity where they can manually add new badges for
distribution within the community, create, access, modify and remove the badges that
they "own" (not the ones that they "have been awarded").
3. Community Users [Access Level I] would have a limited access to the Accolades API
entity using the Liberation entity where they can view the badges received by them
(and by others), access the catalog of available badges, view leaderboards and other
such similar tasks.
Collection interacting with external entities
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. image:: ../_static/badges-proposed-col-ext-interactions.png
:target: ../_images/badges-proposed-col-ext-interactions.png
1. Repository Members [Access Level II] would have complete access to the
Collection entity where they can access, create, modify and remove badge
assets (i.e. artworks, badge rules, awarding conditions etc.). As the
access is unrestricted, they have access to the assets created by someone
else too, which belong to the entity.
1. Repository Members [Access Level II] would have complete access to the Collection
entity where they can access, create, modify and remove badge assets (i.e. artworks,
badge rules, awarding conditions etc.). As the access is unrestricted, they have
access to the assets created by someone else too, which belong to the entity.
Database interacting with external entities
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. image:: ../_static/badges-proposed-dtb-ext-interactions.png
:target: ../_images/badges-proposed-dtb-ext-interactions.png
1. Service Admins [Access Level IV] would have a direct control on the
deployment environments (staging, development, production etc.).
1. Service Admins [Access Level IV] would have a direct control on the deployment
environments (staging, development, production etc.).
Messages Consumer interacting with external entities
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. image:: ../_static/badges-proposed-msg-ext-interactions.png
:target: ../_images/badges-proposed-msg-ext-interactions.png
1. Service Admins [Access Level IV] would have a direct control on the
deployment environments (staging, development, production etc.).
1. Service Admins [Access Level IV] would have a direct control on the deployment
environments (staging, development, production etc.).
External-only interactions
----
--------------------------
Interactions involving only external entities are termed, in this context, as
external-only interactions.
1. Badge Owners [Access Level III] might need to interact with Repository
Members [Access Level II] for assistance with changing badge assets in
the Collection entity.
2. Community User [Access Level I] might need to interact with the Badge
Owners [Access Level III] to obtain badges for the action that they have
done within the community.
1. Badge Owners [Access Level III] might need to interact with Repository Members
[Access Level II] for assistance with changing badge assets in the Collection entity.
2. Community User [Access Level I] might need to interact with the Badge Owners [Access
Level III] to obtain badges for the action that they have done within the community.

55
docs/badges/prop_rewrite_technologies.rst
View file

@ -1,52 +1,57 @@
.. _prop_rewrite_technologies:
Technologies suggested
====
======================
Database
----
--------
*Suggested using:*
| Database: Postgres
| Database: Postgres
Accolades API
----
-------------
*Suggested using:*
| Web framework: `FastAPI <https://fastapi.tiangolo.com/>`_ or `Flask <https://flask.palletsprojects.com/>`_
| CLI compositor: `Click <https://click.palletsprojects.com/>`_
| Database abstraction: `Psycopg2 <https://www.psycopg.org/docs/>`_
| ORM and SQL library: `SQLAlchemy <https://sqlalchemy.org/>`_
| Configuration parsing: `PyYAML <https://pyyaml.org/wiki/PyYAML>`_ or `Tomllib <https://docs.python.org/3/library/tomllib.html>`_
| Caching and indexing: `Redis <https://redis.io/>`_
| Web framework: `FastAPI <https://fastapi.tiangolo.com/>`_ or `Flask
<https://flask.palletsprojects.com/>`_
| CLI compositor: `Click <https://click.palletsprojects.com/>`_
| Database abstraction: `Psycopg2 <https://www.psycopg.org/docs/>`_
| ORM and SQL library: `SQLAlchemy <https://sqlalchemy.org/>`_
| Configuration parsing: `PyYAML <https://pyyaml.org/wiki/PyYAML>`_ or `Tomllib
<https://docs.python.org/3/library/tomllib.html>`_
| Caching and indexing: `Redis <https://redis.io/>`_
Accolades CLI
----
-------------
*Suggested using:*
| CLI compositor: `Click <https://click.palletsprojects.com/>`_
| HTTP library: `Requests <https://requests.readthedocs.io/>`_
| Configuration parsing: `PyYAML <https://pyyaml.org/wiki/PyYAML>`_ or `Tomllib <https://docs.python.org/3/library/tomllib.html>`_
| CLI compositor: `Click <https://click.palletsprojects.com/>`_
| HTTP library: `Requests <https://requests.readthedocs.io/>`_
| Configuration parsing: `PyYAML <https://pyyaml.org/wiki/PyYAML>`_ or `Tomllib
<https://docs.python.org/3/library/tomllib.html>`_
Liberation
----
----------
*Suggested using:*
| Frontend library: `VueJS <https://vuejs.org/>`_
| Frontend framework: `Vuetify <https://vuetifyjs.com/>`_
| Server: `Vue server <https://vuejs.org/>`_
| Frontend library: `VueJS <https://vuejs.org/>`_
| Frontend framework: `Vuetify <https://vuetifyjs.com/>`_
| Server: `Vue server <https://vuejs.org/>`_
Collection
----
----------
*Suggested using:*
| Version controlled forge: `GitLab <https://gitlab.com/>`_
| Version controlled forge: `GitLab <https://gitlab.com/>`_
Messages Consumer
-----
-----------------
*Suggested using:*
| Listener: `Fedora Messaging <https://fedora-messaging.readthedocs.io/en/stable/>`_
| CLI compositor: `Click <https://click.palletsprojects.com/>`_
| HTTP library: `Requests <https://requests.readthedocs.io/>`_
| Configuration parsing: `PyYAML <https://pyyaml.org/wiki/PyYAML>`_ or `Tomllib <https://docs.python.org/3/library/tomllib.html>`_
| Listener: `Fedora Messaging
<https://fedora-messaging.readthedocs.io/en/stable/>`_
| CLI compositor: `Click <https://click.palletsprojects.com/>`_
| HTTP library: `Requests <https://requests.readthedocs.io/>`_
| Configuration parsing: `PyYAML <https://pyyaml.org/wiki/PyYAML>`_ or `Tomllib
<https://docs.python.org/3/library/tomllib.html>`_

69
docs/badges/proposal_revitalize.rst
View file

@ -1,50 +1,45 @@
.. _proposal_revitalize:
Revitalize Fedora Badges
====
========================
| *At this point I don’t think we should put in more effort,*
| *it seems that a complete rewrite is needed.*
| -- **zlopez**
| *At this point I don’t think we should put in more effort,*
| *it seems that a complete rewrite is needed.*
| -- **zlopez**
Why we think revitalizing is not a good idea
------
--------------------------------------------
1. Conversion of tahrir codebase from python2 into python3 would be a huge effort.
Most of the code wasn’t touched in for over 3 years. The development
environment could be set up, but needs plenty of updates. There is almost
no documentation, so this needs to be created as well. The codebase is small,
around 2000 lines of code, but it’s heavily dependent on the pyramid framework,
which doesn’t have a great documentation and we no longer have people with
the knowledge of pyramid in the CPE Team. We also don’t have knowledge in the
team about the mako templates, but they are easy to comprehend.
The CI probably worked in the past (.cico.pipeline file is available),
1. Conversion of tahrir codebase from python2 into python3 would be a huge effort. Most
of the code wasn’t touched in for over 3 years. The development environment could be
set up, but needs plenty of updates. There is almost no documentation, so this needs
to be created as well. The codebase is small, around 2000 lines of code, but it’s
heavily dependent on the pyramid framework, which doesn’t have a great documentation
and we no longer have people with the knowledge of pyramid in the CPE Team. We also
don’t have knowledge in the team about the mako templates, but they are easy to
comprehend. The CI probably worked in the past (.cico.pipeline file is available),
but it’s no longer working, recommending to use either Zuul CI or GitHub Actions.
:What needs to be done:
- Port from python2 to python3
- Fix the dev environment
- Add CI
- Add dependency management (Optional)
- Add tests
- Convert from fedmsg to fedora messaging
- Add documentation
- Convert from Pyramid to Flask (Optional)
- Use poetry (Optional)
:What needs to be done: - Port from python2 to python3
- Fix the dev environment
- Add CI
- Add dependency management (Optional)
- Add tests
- Convert from fedmsg to fedora messaging
- Add documentation
- Convert from Pyramid to Flask (Optional)
- Use poetry (Optional)
2. tahrir_api code is tested against python 3.5, 3.6 and 3.7 and test coverage is 74%.
CI is using github actions. The repo needs some cleaning. There are still
Travis CI and .cico.pipeline config. The database is using SQLAlchemy and
alembic for migration.
Tahrir-api doesn’t have any dev environment where you can test it out.
It has only README as a documentation.
Has no dependency management, I’m recommending renovate.
API is using pyramid as a framework, which brings disadvantages mentioned
CI is using github actions. The repo needs some cleaning. There are still Travis CI
and .cico.pipeline config. The database is using SQLAlchemy and alembic for
migration. Tahrir-api doesn’t have any dev environment where you can test it out. It
has only README as a documentation. Has no dependency management, I’m recommending
renovate. API is using pyramid as a framework, which brings disadvantages mentioned
earlier.
:What needs to be done:
- Convert from Pyramid to Flask (Optional)
- Add some tools to test it out in dev environment
- Add dependency management (Optional)
- Improve documentation
- Use poetry (Optional)
:What needs to be done: - Convert from Pyramid to Flask (Optional)
- Add some tools to test it out in dev environment
- Add dependency management (Optional)
- Improve documentation
- Use poetry (Optional)

92
docs/badges/proposal_rewrite.rst
View file

@ -1,64 +1,64 @@
.. _proposal_rewrite:
Rewrite Fedora Badges From The Ground Up
====
========================================
| *Something which is state-of-the-art right now,*
| *will eventually and inevitably become a tech debt tomorrow.*
| -- **Pierre-Yves Chibbon**, Sometime in 2021
| *Something which is state-of-the-art right now,*
| *will eventually and inevitably become a tech debt tomorrow.*
| -- **Pierre-Yves Chibbon**, Sometime in 2021
We are proposing a rewrite for the project from the ground up.
Justification
----
1. The web interface and the API service for Fedora Badges is written in
Python 2 which has gone `EOL <https://www.python.org/doc/sunset-python-2/>`_
on January 1st, 2020. We could have proposed a 1:1 rewrite of the same
functionality in Python 3 but reimplementing the features in ways that are
more relevant now, makes more sense.
2. The `messages consumer <https://github.com/fedora-infra/fedbadges>`_
is bound to the `fedmsg <https://github.com/fedora-infra/fedmsg>`_ and needs
to be reimplemented to be compatible with
`Fedora Messaging <https://github.com/fedora-infra/fedora-messaging>`_.
As these two projects are inherently different from each other in their
function, not a lot of code can be salvaged from the previous implementation
of the messsages consumer, even though the strategy/approach can be.
3. The web interface and the API service for Fedora Badges is written using
the `Pyramid web framework <https://trypyramid.com/>`_ and makes use of
`Mako templating system <https://www.makotemplates.org/>`_ which are not as
`popular <https://gustavwillig.medium.com/python-web-development-in-2021-which-web-frameworks-are-the-most-popular-by-github-stars-e07b1d7ef6f7>`_
as their alternatives, `Flask <https://flask.palletsprojects.com/>`_ or
`FastAPI <https://fastapi.tiangolo.com/>`_ and
`Jinja <https://jinja.palletsprojects.com/>`_ respectively are.
* Maintaining the codebase in its current state can be a challenge in
the absence of comprehensive documentation, popular community support
and frequent question/answer activity on forums regarding the topics
related to the Pyramid web framework and Mako templating system.
* Finding interested community members (or getting people interested to
contribute to the project) can be difficult if a set of comparatively
obscure or unpopular technologies are used, which might deincentivize
participation here due to a relatively steep learning curve.
4. The frontend of the web interface is written in a form of a meta-application
(i.e. the frontend is supposed to behave like an application and yet is
implemented like a website) using (now, dated) templating systems, plain
HTML, CSS3 frameworks and Javascript libraries. These could use a rewrite
for improved application-like functionality and interactivity.
5. The current codebase has residual and/or partial implementation to
accommodate Open Badges (or Badgr) standards for storing artworks data,
awarding conditions, awardees etc. which have potentially never been
completed and/or deployed. Cleaning up the said codebase or completing it
before maintaining the project in its current state would require lots of
effort.
-------------
1. The web interface and the API service for Fedora Badges is written in Python 2 which
has gone `EOL <https://www.python.org/doc/sunset-python-2/>`_ on January 1st, 2020.
We could have proposed a 1:1 rewrite of the same functionality in Python 3 but
reimplementing the features in ways that are more relevant now, makes more sense.
2. The `messages consumer <https://github.com/fedora-infra/fedbadges>`_ is bound to the
`fedmsg <https://github.com/fedora-infra/fedmsg>`_ and needs to be reimplemented to
be compatible with `Fedora Messaging
<https://github.com/fedora-infra/fedora-messaging>`_. As these two projects are
inherently different from each other in their function, not a lot of code can be
salvaged from the previous implementation of the messsages consumer, even though the
strategy/approach can be.
3. The web interface and the API service for Fedora Badges is written using the `Pyramid
web framework <https://trypyramid.com/>`_ and makes use of `Mako templating system
<https://www.makotemplates.org/>`_ which are not as `popular
<https://gustavwillig.medium.com/python-web-development-in-2021-which-web-frameworks-are-the-most-popular-by-github-stars-e07b1d7ef6f7>`_
as their alternatives, `Flask <https://flask.palletsprojects.com/>`_ or `FastAPI
<https://fastapi.tiangolo.com/>`_ and `Jinja <https://jinja.palletsprojects.com/>`_
respectively are.
- Maintaining the codebase in its current state can be a challenge in the absence of
comprehensive documentation, popular community support and frequent question/answer
activity on forums regarding the topics related to the Pyramid web framework and
Mako templating system.
- Finding interested community members (or getting people interested to contribute to
the project) can be difficult if a set of comparatively obscure or unpopular
technologies are used, which might deincentivize participation here due to a
relatively steep learning curve.
4. The frontend of the web interface is written in a form of a meta-application (i.e.
the frontend is supposed to behave like an application and yet is implemented like a
website) using (now, dated) templating systems, plain HTML, CSS3 frameworks and
Javascript libraries. These could use a rewrite for improved application-like
functionality and interactivity.
5. The current codebase has residual and/or partial implementation to accommodate Open
Badges (or Badgr) standards for storing artworks data, awarding conditions, awardees
etc. which have potentially never been completed and/or deployed. Cleaning up the
said codebase or completing it before maintaining the project in its current state
would require lots of effort.
Implementation
----
--------------
Here is how we are planning to have the project redeveloped.
.. toctree::
:maxdepth: 1
prop_rewrite_entities
prop_rewrite_interactions
prop_rewrite_technologies

39
docs/bodhi/decouple_bodhi_from_pdc.rst
View file

@ -1,42 +1,37 @@
.. _decouple_bodhi_from_pdc:
Decouple Bodhi from pdc
========================================
=======================
Resources
----------
* Dtabase Models: https://bodhi.fedoraproject.org/docs/developer/models.html
---------
- Dtabase Models: https://bodhi.fedoraproject.org/docs/developer/models.html
Discussion
----------
Bodhi currently relies on PDC for criticl path package information, this
information could potentiallt=y be stored in Bodhi itself to remove the need to
call out to PDC at all. PDC currently works but is not maintained, n the case of
a CVE or some other reason we need to update PDC there could be knock on effects
with Bodhi.
Bodhi currently relies on PDC for criticl path package information, this information
could potentiallt=y be stored in Bodhi itself to remove the need to call out to PDC at
all. PDC currently works but is not maintained, n the case of a CVE or some other reason
we need to update PDC there could be knock on effects with Bodhi.
What would be needed
--------------------
To add this functionality to Bodhi we would need to investigate if the packaging
entity present has the ability to cope with this change. New tooling would need
to be added to Bodhi to cope with the changes in the application.
The biggest piece of work would be to remodel the database to include a new
relation which would have the information to check if each package is critical
path for each release. Every package would need a record to specify if it was
critical path or not.
To add this functionality to Bodhi we would need to investigate if the packaging entity
present has the ability to cope with this change. New tooling would need to be added to
Bodhi to cope with the changes in the application.
The biggest piece of work would be to remodel the database to include a new relation
which would have the information to check if each package is critical path for each
release. Every package would need a record to specify if it was critical path or not.
Conclusion
----------
After some initial investigation and discussion it was deemed that this should
not go into bodhi.
After some initial investigation and discussion it was deemed that this should not go
into bodhi.
Dist-git is the source of truth for al packages so all package information
should be held in that application
Dist-git is the source of truth for al packages so all package information should be
held in that application

39
docs/bodhi/index.rst
View file

@ -1,20 +1,20 @@
Bodhi
========
=====
Purpose
-------
This investigation is intended to investigate possible improvements to Bodhi.
The suggested improvement areas are related to improving packaging to allow for
smaller incremental releases of bodhi, fix unit and integration tests in CI,
simplify and partially automate the release process, add dependency managment
and to decouple Bodhi from PDC
This investigation is intended to investigate possible improvements to Bodhi. The
suggested improvement areas are related to improving packaging to allow for smaller
incremental releases of bodhi, fix unit and integration tests in CI, simplify and
partially automate the release process, add dependency managment and to decouple Bodhi
from PDC
Resources
---------
* Bodhi repo: https://github.com/fedora-infra/bodhi
* Bodhi wiki: https://fedoraproject.org/wiki/Bodhi
- Bodhi repo: https://github.com/fedora-infra/bodhi
- Bodhi wiki: https://fedoraproject.org/wiki/Bodhi
Investigation
-------------
@ -27,24 +27,23 @@ Here is the list of ideas/things we discussed/looked at:
separate_bodhi_packages
decouple_bodhi_from_pdc
Conclusions
-----------
There was a number of separate improvements to Bodhi proposed for this
investigation some of which are needed and can be carried out as part of general
Infra & Releng work under mini-initiatives.
There was a number of separate improvements to Bodhi proposed for this investigation
some of which are needed and can be carried out as part of general Infra & Releng work
under mini-initiatives.
The investigation also concluded that Bodhi does not need to take over any
functionality from PDC. As dist-git is the source of truth for packages all the
information relating to packages from PDC should instead be moved there.
The investigation also concluded that Bodhi does not need to take over any functionality
from PDC. As dist-git is the source of truth for packages all the information relating
to packages from PDC should instead be moved there.
Proposed Roadmap
----------------
* Mini-initiative to separate Bodhi into multiple sub packages
- Mini-initiative to separate Bodhi into multiple sub packages
https://pagure.io/fedora-infrastructure/issue/10368
* Ticket to fix integration and unit tests in CI
* Ticket to fix dependency management in Bodhi
* Ticket to automat part of the release process
* Propose dist-git changes
- Ticket to fix integration and unit tests in CI
- Ticket to fix dependency management in Bodhi
- Ticket to automat part of the release process
- Propose dist-git changes

32
docs/bodhi/separate_bodhi_packages.rst
View file

@ -1,35 +1,31 @@
.. _separate_bodhi_packages:
Split Bodhi Into Separate Python Packages
=========================================
Resources
---------
* Bodhi source code: https://github.com/fedora-infra/bodhi
* Python packaging docs: https://packaging.python.org/guides/packaging-namespace-packages/
* mini-initiative: https://pagure.io/fedora-infrastructure/issue/10368
- Bodhi source code: https://github.com/fedora-infra/bodhi
- Python packaging docs:
https://packaging.python.org/guides/packaging-namespace-packages/
- mini-initiative: https://pagure.io/fedora-infrastructure/issue/10368
Discussion
----------
Currently Bodhi is a number of packages combined. It may be beneficial to split
into its separate parts. This would allow for each package to be updated and
maintained at its own pace allowing for incremental improvements to bodhi. When
implemented a single part of Bodhi could be deployed at a time instead of having
to deploy the whole application in one go.
The main benefits of this would be an easier to manage application, an easier to
deploy application and a more regularly deployed application as a single issue
fix could be deployed at a time.
Currently Bodhi is a number of packages combined. It may be beneficial to split into its
separate parts. This would allow for each package to be updated and maintained at its
own pace allowing for incremental improvements to bodhi. When implemented a single part
of Bodhi could be deployed at a time instead of having to deploy the whole application
in one go.
The main benefits of this would be an easier to manage application, an easier to deploy
application and a more regularly deployed application as a single issue fix could be
deployed at a time.
Conclusion
----------
This is work that would be of great benefit to the team but is not initiative
worthy on its own. Therefore a mini-initiative seems the best solution.
This is work that would be of great benefit to the team but is not initiative worthy on
its own. Therefore a mini-initiative seems the best solution.

82
docs/communishift/authentication.rst
View file

@ -2,68 +2,68 @@ Authentication in Communishift
==============================
Resources
*********
* https://docs.fedoraproject.org/en-US/infra/ocp4/sop_configure_oauth_ipa/
* https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/ipsilon/#_create_openid_connect_secrets_
---------
- https://docs.fedoraproject.org/en-US/infra/ocp4/sop_configure_oauth_ipa/
- https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/ipsilon/#_create_openid_connect_secrets_
Discussion
**********
Would it be possible to have groups in Openshift linked to groups in FAS. Having
a central place to control group access would make authentication easier and more
----------
Would it be possible to have groups in Openshift linked to groups in FAS. Having a
central place to control group access would make authentication easier and more
transparent.
Identity provider
**********
-----------------
The cluster was linked to the Fedora account system as a necessary precursor to
the investigation.
The cluster was linked to the Fedora account system as a necessary precursor to the
investigation.
An openid secret was created in the private ansible repo using the `openid connect SOP https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/ipsilon/#_create_openid_connect_secrets_`
An openid secret was created in the private ansible repo using the `openid connect SOP
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/ipsilon/#_create_openid_connect_secrets_`
The ipsilon playbook was then run to provision the secret.
To configure openshift this `SOP <https://docs.fedoraproject.org/en-US/infra/ocp4/sop_configure_oauth_ipa/>`_ was follwed to add an oauth client
to allow access to the cluster.
To configure openshift this `SOP
<https://docs.fedoraproject.org/en-US/infra/ocp4/sop_configure_oauth_ipa/>`_ was follwed
to add an oauth client to allow access to the cluster.
Group access
************
------------
Ideally we would like to be able to map groups from IPA to Communishift as this
would make adding and removeing users from projects easier to manage.
Ideally we would like to be able to map groups from IPA to Communishift as this would
make adding and removeing users from projects easier to manage.
Openshift supports group integration with ldap servers, ipa is an ldap based
server however it was deemed not secure to allow an external application have
access to our internal ipa server.
Openshift also supports group mapoping from openid clients which would be the
preffered course of action for us as we are using ipsilon anyway. However this
is not yet supported in Openshift Dedicated. OSD support have said there is an
RFE for this to be added but the ticket is private internally so we cannot track
progress on it.
Openshift supports group integration with ldap servers, ipa is an ldap based server
however it was deemed not secure to allow an external application have access to our
internal ipa server.
Openshift also supports group mapoping from openid clients which would be the preffered
course of action for us as we are using ipsilon anyway. However this is not yet
supported in Openshift Dedicated. OSD support have said there is an RFE for this to be
added but the ticket is private internally so we cannot track progress on it.
Conclusion
******************
----------
As the supported solutions are not suitable it would be necessary to create a
custom solution to carry out group mappings. This could be in the form of an
openshift operator, a toddler or an ansible script run on a cron job.
As the supported solutions are not suitable it would be necessary to create a custom
solution to carry out group mappings. This could be in the form of an openshift
operator, a toddler or an ansible script run on a cron job.
Namespaced groups would need to be created in IPA such as communishift-<project>
and the users added with a sponsor for each group. These would then need to be
automatically replicated in Communishift
Namespaced groups would need to be created in IPA such as communishift-<project> and the
users added with a sponsor for each group. These would then need to be automatically
replicated in Communishift
A possible skeleton solution would be to:
* Periodically call fasjson for any group that begins with communishift-
(but not **communishift** as this already exists and is separate).
* Get the list of users for that group
* Check if the group exixts in openshift and create if not
* Check the list of users in fasjson against the list in Openshift and add/remove
if necessary.
- Periodically call fasjson for any group that begins with communishift- (but not
**communishift** as this already exists and is separate).
- Get the list of users for that group
- Check if the group exixts in openshift and create if not
- Check the list of users in fasjson against the list in Openshift and add/remove if
necessary.
Optional:
* Get the list of sponsors for the group in fasjson and use these to set rbac
permission levels of admin and all other members of the group have basic user access
- Get the list of sponsors for the group in fasjson and use these to set rbac permission
levels of admin and all other members of the group have basic user access

93
docs/communishift/index.rst
View file

@ -2,7 +2,7 @@ Communishift
============
Purpose
*******
-------
Investigate what is needed in order to run a community focused Openshift instance.
@ -10,13 +10,13 @@ Identify possible bottlenecks, issues and whatever the CPE team needs to develop
components/services for this new Openshift instance.
Resources
*********
---------
* https://docs.openshift.com/dedicated/storage/persistent_storage/osd-persistent-storage-aws.html
* https://docs.openshift.com/container-platform/4.6/applications/quotas/quotas-setting-per-project.html
- https://docs.openshift.com/dedicated/storage/persistent_storage/osd-persistent-storage-aws.html
- https://docs.openshift.com/container-platform/4.6/applications/quotas/quotas-setting-per-project.html
Investigation
*************
-------------
The team discussed the following topics:
@ -27,66 +27,75 @@ The team discussed the following topics:
resource-quota
storage
Conclusions
***********
-----------
* The cluster can leverage EFS to provision volumes (using the AWS EFS operator from the Operator Marketplace) and an extra
ansible playbook to automate part of the process;
* Quotas can be enforced by creating an Openshift operator that watches all user namespaces;
* Authentication groups can be automatically synched between FasJSON and Openshift with a new Operator.
- The cluster can leverage EFS to provision volumes (using the AWS EFS operator from the
Operator Marketplace) and an extra ansible playbook to automate part of the process;
- Quotas can be enforced by creating an Openshift operator that watches all user
namespaces;
- Authentication groups can be automatically synched between FasJSON and Openshift with
a new Operator.
Proposed Roadmap
****************
----------------
AWS EFS Ansible Playbook
------------------------
~~~~~~~~~~~~~~~~~~~~~~~~
One needs to provide some AWS info when creating a volume using the EFS operator, sample resource bellow::
One needs to provide some AWS info when creating a volume using the EFS operator, sample
resource bellow:
apiVersion: aws-efs.managed.openshift.io/v1alpha1
kind: SharedVolume
metadata:
name: sv1
namespace: default
spec:
accessPointID: fsap-0123456789abcdef
fileSystemID: fs-0123cdef
.. code-block::
Both "accessPointID and fileSystemID" are generated by AWS with "accessPointID" being generated for every PVC that gets provisioned in the cluster.
apiVersion: aws-efs.managed.openshift.io/v1alpha1
kind: SharedVolume
metadata:
name: sv1
namespace: default
spec:
accessPointID: fsap-0123456789abcdef
fileSystemID: fs-0123cdef
An ansible playbook comes into play to automate the process of creating an "accessPoint" for a namespace whichs should be request in an
infra ticket when requesting the creation of a new namespace in the cluster.
Both "accessPointID and fileSystemID" are generated by AWS with "accessPointID" being
generated for every PVC that gets provisioned in the cluster.
An ansible playbook comes into play to automate the process of creating an "accessPoint"
for a namespace whichs should be request in an infra ticket when requesting the creation
of a new namespace in the cluster.
Fedora Cloud Quota Operator
---------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~
An operator can be created to ensure a namespace's resource quota.
The operator would watch for namespaces with specific tags/annotations (TBD) and apply the required quotas in those namespaces.
The operator would watch for namespaces with specific tags/annotations (TBD) and apply
the required quotas in those namespaces.
The quotas themselves are applied by creating a `ResourceQuota` object
in the namespace is supposed to manage: https://docs.openshift.com/container-platform/4.6/applications/quotas/quotas-setting-per-project.html.
The quotas themselves are applied by creating a `ResourceQuota` object in the namespace
is supposed to manage:
https://docs.openshift.com/container-platform/4.6/applications/quotas/quotas-setting-per-project.html.
Fedora FasJSON Sync Operator
----------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An operator can be used to ensure FasJSON groups are synched with the cluster groups used by Openshift roles.
An operator can be used to ensure FasJSON groups are synched with the cluster groups
used by Openshift roles.
This operator would retrieve a group(s) information every N seconds from FasJSON and apply the changes in the cluster,
ensuring syncrhonization between the two systems.
This operator would retrieve a group(s) information every N seconds from FasJSON and
apply the changes in the cluster, ensuring syncrhonization between the two systems.
Food for thought: it would be interesting if FasJSON notifies group changes to fedora-messaging.
Food for thought: it would be interesting if FasJSON notifies group changes to
fedora-messaging.
Team and Skills
***************
---------------
A team of three indivduals should be able to deliver the proposed roadmap in ~6 weeks (2 week sprint, a sprint per component)
assuming the following technical skills:
A team of three indivduals should be able to deliver the proposed roadmap in ~6 weeks (2
week sprint, a sprint per component) assuming the following technical skills:
* Kubernetes basic concepts/usage;
* API or previous operator knowledge is a plus;
* Ansible basic usage;
* AWS API knowledge is a plus.
- Kubernetes basic concepts/usage; * API or previous operator knowledge is a plus;
- Ansible basic usage; * AWS API knowledge is a plus.
It might be a good opportunity to learn about Kubernetes and Operator/Controller development.
It might be a good opportunity to learn about Kubernetes and Operator/Controller
development.

87
docs/communishift/resource-quota.rst
View file

@ -4,7 +4,7 @@ Resource Quota
Resources
---------
* https://docs.openshift.com/container-platform/4.6/applications/quotas/quotas-setting-per-project.html
- https://docs.openshift.com/container-platform/4.6/applications/quotas/quotas-setting-per-project.html
Discussion
----------
@ -14,51 +14,56 @@ How to limit resource usage per namespace, such as memory, storage and so on.
What would be needed
--------------------
A ResourceQuota object needs to be created in the namespace it is supposed to manage/control.
A ResourceQuota object needs to be created in the namespace it is supposed to
manage/control.
This object could be automatically managed by an operator for each new namespace
that gets created (properly tagged) for community users.
This object could be automatically managed by an operator for each new namespace that
gets created (properly tagged) for community users.
Limits can go from storage, memory and cpu usage to the amount of objects (limit the namespace to have a max. of 5 secrets for example).
Limits can go from storage, memory and cpu usage to the amount of objects (limit the
namespace to have a max. of 5 secrets for example).
Sample object definition::
Sample object definition:
apiVersion: v1
kind: ResourceQuota
metadata:
name: app-quota
spec:
hard:
# compute
cpu: "1" # requests.cpu
memory: "1Gi" # requests.memory
ephemeral-storage: "10Gi" # requests.ephemeral-storage
limits.cpu: "2"
limits.memory: "2Gi"
limits.ephemeral-storage: "10Gi"
# storage
requests.storage: "10Gi"
persistentvolumeclaims: "1"
# <storage-class-name>.storageclass.storage.k8s.io/requests.storage
# <storage-class-name>.storageclass.storage.k8s.io/persistentvolumeclaims
# object counts
pods: "1"
replicationcontrollers: 1
# resourcequotas: 1
# services: 1
# services.loadbalancers: 1
# services.nodeports: 1
# secrets: 1
# configmaps: 1
# openshift.io/imagestreams: 1
# scopes:
# https://docs.openshift.com/container-platform/4.6/applications/quotas/quotas-setting-per-project.html#quotas-scopes_quotas-setting-per-project
# - Terminating
# - NotTerminating
# - BestEffort
# - NotBestEffort
.. code-block::
apiVersion: v1
kind: ResourceQuota
metadata:
name: app-quota
spec:
hard:
# compute
cpu: "1" # requests.cpu
memory: "1Gi" # requests.memory
ephemeral-storage: "10Gi" # requests.ephemeral-storage
limits.cpu: "2"
limits.memory: "2Gi"
limits.ephemeral-storage: "10Gi"
# storage
requests.storage: "10Gi"
persistentvolumeclaims: "1"
# <storage-class-name>.storageclass.storage.k8s.io/requests.storage
# <storage-class-name>.storageclass.storage.k8s.io/persistentvolumeclaims
# object counts
pods: "1"
replicationcontrollers: 1
# resourcequotas: 1
# services: 1
# services.loadbalancers: 1
# services.nodeports: 1
# secrets: 1
# configmaps: 1
# openshift.io/imagestreams: 1
# scopes:
# https://docs.openshift.com/container-platform/4.6/applications/quotas/quotas-setting-per-project.html#quotas-scopes_quotas-setting-per-project
# - Terminating
# - NotTerminating
# - BestEffort
# - NotBestEffort
Conclusion
----------
It can be easily achieved by creating a namespaced resourced and can be automated with an Openshift Operator.
It can be easily achieved by creating a namespaced resourced and can be automated with
an Openshift Operator.

25
docs/communishift/storage.rst
View file

@ -4,28 +4,35 @@ Storage
Resources
---------
* https://docs.openshift.com/dedicated/storage/persistent_storage/osd-persistent-storage-aws.html
- https://docs.openshift.com/dedicated/storage/persistent_storage/osd-persistent-storage-aws.html
Discussion
----------
Find an Openshift storage backend solution so applications can use persistent volumes/storage when needed.
Find an Openshift storage backend solution so applications can use persistent
volumes/storage when needed.
What would be needed
--------------------
The AWS EFS operator can be installed from the Operator Marketplace to provision volumes in Openshift.
The AWS EFS operator can be installed from the Operator Marketplace to provision volumes
in Openshift.
There is a problem where each volume requires an access point to be created in a file system in AWS, this is a manual process.
There is a problem where each volume requires an access point to be created in a file
system in AWS, this is a manual process.
This process can be automated with an ansible playbook as each PVC object will need its own access point but requesting
storage for a namespace can be asked through an infra ticket.
This process can be automated with an ansible playbook as each PVC object will need its
own access point but requesting storage for a namespace can be asked through an infra
ticket.
AWS does not apply any limits to the created volume that control needs to be managed in Openshift.
AWS does not apply any limits to the created volume that control needs to be managed in
Openshift.
Conclusion
----------
The AWS EFS operator is the most straightforward path to support persistent storage in Communishift.
The AWS EFS operator is the most straightforward path to support persistent storage in
Communishift.
There is a manual step which requires the creation of an access point in AWS but that can be automated with ansible.
There is a manual step which requires the creation of an access point in AWS but that
can be automated with ansible.

24
docs/datanommer_datagrepper/datanommer.rst
View file

@ -3,15 +3,15 @@
Datanommer
==========
* Reads-in messages from the bus
* Stores them into the database
- Reads-in messages from the bus
- Stores them into the database
Database tables
---------------
Here is how the database schema looks like currently:
::
.. code-block::
datanommer=# \dt
List of relations
@ -24,14 +24,12 @@ Here is how the database schema looks like currently:
public | user | table | datanommer
public | user_messages | table | datanommer
Table sizes
-----------
Here is the size of each table:
::
.. code-block::
datanommer-#
SELECT
@ -49,16 +47,15 @@ Here is the size of each table:
alembic_version | 8192 bytes | 0 bytes
(6 rows)
The 3 columns are:
The 3 columns are::
.. code-block::
Table – The name of the table
Size – The total size that this table takes
External Size – The size that related objects of this table like indices take
::
.. code-block::
datanommer=#
SELECT
@ -109,12 +106,15 @@ The 3 columns are::
sql_features | r | 716 | 64 kB
(37 rows)
The 4 columns are:
The 4 columns are::
.. code-block::
objectname – The name of the object
objecttype – r for the table, i for an index, t for toast data, ...
#entries – The number of entries in the object (e.g. rows)
size – The size of the object
(source for these queries: https://wiki-bsse.ethz.ch/display/ITDOC/Check+size+of+tables+and+objects+in+PostgreSQL+database )
(source for these queries:
https://wiki-bsse.ethz.ch/display/ITDOC/Check+size+of+tables+and+objects+in+PostgreSQL+database
)

21
docs/datanommer_datagrepper/default_delta.rst
View file

@ -1,18 +1,17 @@
Default delta
=============
Checking the current status of datagrepper, we realized that not specifying a
`delta` value in the URL led to timeouts while specifying one, makes datagrepper
return properly.
Checking the current status of datagrepper, we realized that not specifying a `delta`
value in the URL led to timeouts while specifying one, makes datagrepper return
properly.
Investigating the configuration options of datagrepper, we found out that
there is a `DEFAULT_QUERY_DELTA` configuration key that allows to specify a
default delta value when one is not specified.
Investigating the configuration options of datagrepper, we found out that there is a
`DEFAULT_QUERY_DELTA` configuration key that allows to specify a default delta value
when one is not specified.
Just setting that configuration key to ``60*60*24*3`` (ie: 3 days) improves the
datagrepper performances quite a bit (as in queries actually return instead of
timing out).
datagrepper performances quite a bit (as in queries actually return instead of timing
out).
That configuration change, does break the API a little bit as with it, it will
limit the messages returned to the last 3 days.
That configuration change, does break the API a little bit as with it, it will limit the
messages returned to the last 3 days.

93
docs/datanommer_datagrepper/index.rst
View file

@ -4,21 +4,18 @@ Datanommer / Datagrepper
Datanommer
----------
* Reads-in messages from the bus
* Stores them into the database
- Reads-in messages from the bus
- Stores them into the database
.. toctree::
:maxdepth: 1
datanommer
Datagrepper
-----------
* Exposes the messages in the database via an API with different filtering
capacity
- Exposes the messages in the database via an API with different filtering capacity
Investigation
-------------
@ -35,49 +32,43 @@ Here is the list of ideas/things we looked at:
pg_array_column_postgrest
stats
Conclusions
-----------
We have investigated different ways to improve the database storing our 180
millions messages. While we considered looking at the datagrepper application
itself as well, we considered that replacing datagrepper with another application
would have too large consequences. We have a number of applications in our
realm that rely on datagrepper's API and there is an unknown number of applications
outside our realm that make use of it as well.
Breaking all of these applications is a non-goal for us. For this reason we
We have investigated different ways to improve the database storing our 180 millions
messages. While we considered looking at the datagrepper application itself as well, we
considered that replacing datagrepper with another application would have too large
consequences. We have a number of applications in our realm that rely on datagrepper's
API and there is an unknown number of applications outside our realm that make use of it
as well. Breaking all of these applications is a non-goal for us. For this reason we
focused on postgresql first.
We looked at different solutions, starting with manually partitioning on year,
then on ``id`` (not ``msg_id``, the primary key field ``id`` which is an integer).
We then looked at using the postgresql plugin `timescaledb` and finally we looked
at using this plugin together with a database model change where the relation
tables are merged into the main ``messages`` table and their is stored using
arrays.
We looked at different solutions, starting with manually partitioning on year, then on
``id`` (not ``msg_id``, the primary key field ``id`` which is an integer). We then
looked at using the postgresql plugin `timescaledb` and finally we looked at using this
plugin together with a database model change where the relation tables are merged into
the main ``messages`` table and their is stored using arrays.
Based on our investigations, our recommendation is to migrate the postgresql
database to use the `timescaledb` plugin and configure datagrepper to have a
default delta value via ``DEFAULT_QUERY_DELTA``.
Based on our investigations, our recommendation is to migrate the postgresql database to
use the `timescaledb` plugin and configure datagrepper to have a default delta value via
``DEFAULT_QUERY_DELTA``.
As a picture is worth a thousand words:
.. image:: ../_static/datanommer_percent_sucess.jpg
:target: ../_images/datanommer_percent_sucess.jpg
We checked, setting a ``DEFAULT_QUERY_DELTA`` alone provides already some
performance gain, using `timescaledb` with ``DEFAULT_QUERY_DELTA`` provide the
most gain but using `timescaledb` without ``DEFAULT_QUERY_DELTA`` brings back
the time out issues we are seeing today when datagrepper is queried without a
specified ``delta`` value.
We checked, setting a ``DEFAULT_QUERY_DELTA`` alone provides already some performance
gain, using `timescaledb` with ``DEFAULT_QUERY_DELTA`` provide the most gain but using
`timescaledb` without ``DEFAULT_QUERY_DELTA`` brings back the time out issues we are
seeing today when datagrepper is queried without a specified ``delta`` value.
We also believe that the performance gain observed with `timescaledb` could be
reproduced if we were to do the partitioning ourself on the ``timestamp`` field
of the ``messages`` table. However, it would mean that we have to manually
maintain that partitioning, take care of creating the new partitions as needed
and so on, while `timescaledb` provides all of this for us automatically, thus
simplifying the long term maintenance of that database.
reproduced if we were to do the partitioning ourself on the ``timestamp`` field of the
``messages`` table. However, it would mean that we have to manually maintain that
partitioning, take care of creating the new partitions as needed and so on, while
`timescaledb` provides all of this for us automatically, thus simplifying the long term
maintenance of that database.
Proposed roadmap
~~~~~~~~~~~~~~~~
@ -86,40 +77,34 @@ We propose the following roadmap to improve datanommer and datagrepper:
0/ Announce the upcoming API breakage and outage of datagrepper
Be loud about the upcoming changes and explain how the API breakage can be
mitigated.
Be loud about the upcoming changes and explain how the API breakage can be mitigated.
1/ Port datanommer to fedora-messaging and openshift
This will ensure that there are no duplicate messages are saved in the database
(cf our ref:`timescaledb_findings`).
It will also provide a way to store the messages while datagrepper is being
upgraded (which will require an outage). Using lazy queues in rabbitmq may be
a way to store the high number of messages that will pile up during the outage
window (which will be over 24h).
This will ensure that there are no duplicate messages are saved in the database (cf our
ref:`timescaledb_findings`). It will also provide a way to store the messages while
datagrepper is being upgraded (which will require an outage). Using lazy queues in
rabbitmq may be a way to store the high number of messages that will pile up during the
outage window (which will be over 24h).
Rabbitmq lazy queues: https://www.rabbitmq.com/lazy-queues.html
2/ Port datagrepper to timescaledb.
This will improve the performance of the UI. Thanks to rabbitmq, no messages will
be lost, they will only show up in datagrepper at the end of the outage and
with a delayed timestamp.
This will improve the performance of the UI. Thanks to rabbitmq, no messages will be
lost, they will only show up in datagrepper at the end of the outage and with a delayed
timestamp.
3/ Configure datagrepper to have a ``DEFAULT_QUERY_DELTA``.
This will simply bound a number of queries which otherwise run slow and lead to
timeouts at the application level.
This will simply bound a number of queries which otherwise run slow and lead to timeouts
at the application level.
4/ Port datagrepper to openshift
This will make it easier to maintain and/or scale as needed.
5/ Port datagrepper to fedora-messaging
This will allow to make use of the fedora-messaging schemas provided by the
applications instead of relying on `fedmsg_meta_fedora_infrastructure`.
This will allow to make use of the fedora-messaging schemas provided by the applications
instead of relying on `fedmsg_meta_fedora_infrastructure`.

104
docs/datanommer_datagrepper/pg_array_column_postgrest.rst
View file

@ -4,64 +4,74 @@ Using the array type for user and package queries
Currently, we use auxiliary tables to query for messages related to packages or users,
in the standard RDBS fashion.
We came to some problems when trying to enforce foreign key constrains while using the timescaledb
extension. We decided to try, if just using a column with array type with proper indes would have simmilar performace.
We came to some problems when trying to enforce foreign key constrains while using the
timescaledb extension. We decided to try, if just using a column with array type with
proper indes would have simmilar performace.
Array columns support indexing with Generalized Inverted Index, GIN,
that allows for fast searches on membership and intersection. Because we mostly search for memebership,
Array columns support indexing with Generalized Inverted Index, GIN, that allows for
fast searches on membership and intersection. Because we mostly search for memebership,
array column could be performant enough for our purposes.
Resources
---------
* PG 12 Array type: https://www.postgresql.org/docs/12/arrays.html
* GIN index: https://www.postgresql.org/docs/12/gin.html
* GIN operators for BTREE: https://www.postgresql.org/docs/11/btree-gin.html
- PG 12 Array type: https://www.postgresql.org/docs/12/arrays.html
- GIN index: https://www.postgresql.org/docs/12/gin.html
- GIN operators for BTREE: https://www.postgresql.org/docs/11/btree-gin.html
Installing/enabling/activating
------------------------------
To have comparable results, we enabled timescaledb in same fashion as in our other experiment.
To have comparable results, we enabled timescaledb in same fashion as in our other
experiment.
To add new column
::
alter table messages2 add column packages text[];
.. code-block::
alter table messages2 add column packages text[];
To populate it
::
update messages2 set packages=t_agg.p_agg from
(select msg, array_agg(package) as p_agg from package_messages group by msg) as t_agg where messages.id=t_agg.msg;
.. code-block::
We need to enable the btree_gin extension to be able to create index with array as well as timestamp
::
update messages2 set packages=t_agg.p_agg from
(select msg, array_agg(package) as p_agg from package_messages group by msg) as t_agg where messages.id=t_agg.msg;
CREATE EXTENSION btree_gin;
We need to enable the btree_gin extension to be able to create index with array as well
as timestamp
.. code-block::
CREATE EXTENSION btree_gin;
To create the index
::
.. code-block::
CREATE INDEX idx_msg_user on "messages2" USING GIN ("timestamp", "packages");
To help reuse our testing script, we setup postgrest locally
::
podman run --rm --net=host -p 3000:3000 -e PGRST_DB_URI=$DBURI -e PGRST_DB_ANON_ROLE="datagrepper" -e PGRST_MAX_ROWS=25 postgrest/postgrest:v7.0.
.. code-block::
Because we focused only on package queries, as user colun couldn't be populated due to constraints on size,
we chose two as representative. There is implicit limit to return just 25 rows.
podman run --rm --net=host -p 3000:3000 -e PGRST_DB_URI=$DBURI -e PGRST_DB_ANON_ROLE="datagrepper" -e PGRST_MAX_ROWS=25 postgrest/postgrest:v7.0.
Because we focused only on package queries, as user colun couldn't be populated due to
constraints on size, we chose two as representative. There is implicit limit to return
just 25 rows.
A simple membership:
::
/messages_ts?packages=ov.{{kernel}}
.. code-block::
/messages_ts?packages=ov.{{kernel}}
A simple membership ordered by time.
::
/messages_ts?order=timestamp.desc&packages=ov.{{kernel}}
.. code-block::
/messages_ts?order=timestamp.desc&packages=ov.{{kernel}}
Findings
--------
@ -69,36 +79,40 @@ Findings
Querying just the package membership
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The queries were surprisingly fast, with maximum under 4 seconds and
mean around half a second. This encouraged us to do further experiments.
The queries were surprisingly fast, with maximum under 4 seconds and mean around half a
second. This encouraged us to do further experiments.
Results ::
Results
test_filter_by_package
Requests: 300, pass: 300, fail: 0, exception: 0
For pass requests:
Request per Second - mean: 3.63
Time per Request - mean: 0.522946, min: 0.000000, max: 3.907548
.. code-block::
test_filter_by_package
Requests: 300, pass: 300, fail: 0, exception: 0
For pass requests:
Request per Second - mean: 3.63
Time per Request - mean: 0.522946, min: 0.000000, max: 3.907548
Querying just the package membership ordered by timestamp desc
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Usually we want to see most recent messages. So we ammended the query,
to include "order by timestamp desc". The result was less encouraging,
with longest succesful query taking more than 90 seconds and several timing out.
Usually we want to see most recent messages. So we ammended the query, to include "order
by timestamp desc". The result was less encouraging, with longest succesful query taking
more than 90 seconds and several timing out.
This seems to be the result of GIN index not supporting order in the index.
Results ::
Results
test_filter_by_package
Requests: 300, pass: 280, fail: 0, exception: 20
For pass requests:
Request per Second - mean: 0.53
Time per Request - mean: 7.474040, min: 0.000000, max: 99.880939
.. code-block::
test_filter_by_package
Requests: 300, pass: 280, fail: 0, exception: 20
For pass requests:
Request per Second - mean: 0.53
Time per Request - mean: 7.474040, min: 0.000000, max: 99.880939
Conclusion
----------
While array support seems interesting, and for simple queries very fast, indexes that require ordering
don't seem to be supported. This makes strong case against using them.
While array support seems interesting, and for simple queries very fast, indexes that
require ordering don't seem to be supported. This makes strong case against using them.

65
docs/datanommer_datagrepper/pg_partitioning.rst
View file

@ -1,60 +1,55 @@
Partitioning the database
=========================
In the database used by datanommer and datagrepper one table stands out from the
other ones by its size, the ``messages`` table. This can be observed in
:ref:`datanommer`.
In the database used by datanommer and datagrepper one table stands out from the other
ones by its size, the ``messages`` table. This can be observed in :ref:`datanommer`.
One possibility to speed things up in datagrepper is to partition that table
into a set of smaller sized partitions.
One possibility to speed things up in datagrepper is to partition that table into a set
of smaller sized partitions.
Here are some resources regarding partitioning postgresql tables:
* Table partitioning at postgresql's documentation: https://www.postgresql.org/docs/13/ddl-partitioning.html
* How to use table partitioning to scale PostgreSQL: https://www.enterprisedb.com/postgres-tutorials/how-use-table-partitioning-scale-postgresql
* Definition of PostgreSQL Partition: https://www.educba.com/postgresql-partition/
- Table partitioning at postgresql's documentation:
https://www.postgresql.org/docs/13/ddl-partitioning.html
- How to use table partitioning to scale PostgreSQL:
https://www.enterprisedb.com/postgres-tutorials/how-use-table-partitioning-scale-postgresql
- Definition of PostgreSQL Partition: https://www.educba.com/postgresql-partition/
Attempt #1
----------
For our first attempt at partitioning the `messages` table, we thought we would
partition it by year. Having a different partition for each year.
We thus started by adding a ``year`` field to the table and fill it by extracting
the year from the ``timestamp`` field of the table.
partition it by year. Having a different partition for each year. We thus started by
adding a ``year`` field to the table and fill it by extracting the year from the
``timestamp`` field of the table.
However, one thing to realize when using partitioned table is that each partition
needs to be considered as an independant table. Meaning an unique constraint has
to involve the field on which the table is partitioned.
In other words, if you partition the table by a year field, that year field will
need to be part of the primary key as well as any ``UNIQUE`` constraint on the
table.
So to partition the `messages` table on ``year``, we had to add the ``year``
field to the primary key. However, that broke the foreign key constraints on
the ``user_messages`` and ``package_messages`` tables which rely on the ``id``
field to link the tables.
However, one thing to realize when using partitioned table is that each partition needs
to be considered as an independant table. Meaning an unique constraint has to involve
the field on which the table is partitioned. In other words, if you partition the table
by a year field, that year field will need to be part of the primary key as well as any
``UNIQUE`` constraint on the table.
So to partition the `messages` table on ``year``, we had to add the ``year`` field to
the primary key. However, that broke the foreign key constraints on the
``user_messages`` and ``package_messages`` tables which rely on the ``id`` field to link
the tables.
Attempt #2
----------
Since partitioning on ``year`` did not work, we reconsidered and decided to
partition on the ``id`` field instead using `RANGE PARTITION`.
We partitioned the ``messages`` table on the ``id`` field with partition of 10
million records each. This has the advantage of making each partition of similar
sizes.
Since partitioning on ``year`` did not work, we reconsidered and decided to partition on
the ``id`` field instead using `RANGE PARTITION`.
We partitioned the ``messages`` table on the ``id`` field with partition of 10 million
records each. This has the advantage of making each partition of similar sizes.
More resources
--------------
These are a few more resources we looked at and thought were worth bookmarking:
* Automatic partitioning by day - PostgreSQL: https://stackoverflow.com/questions/55642326/
* pg_partman, partition manager: https://github.com/pgpartman/pg_partman
* How to scale PostgreSQL 10 using table inheritance and declarative partitioning: https://blog.timescale.com/blog/scaling-partitioning-data-postgresql-10-explained-cd48a712a9a1/
- Automatic partitioning by day - PostgreSQL:
https://stackoverflow.com/questions/55642326/
- pg_partman, partition manager: https://github.com/pgpartman/pg_partman
- How to scale PostgreSQL 10 using table inheritance and declarative partitioning:
https://blog.timescale.com/blog/scaling-partitioning-data-postgresql-10-explained-cd48a712a9a1/

16
docs/datanommer_datagrepper/pg_stat_statements.rst
View file

@ -1,14 +1,16 @@
Postgresql's pg_stat_statements
===============================
This is a postgresql module allowing to track planning and execution statistics
of all SQL statements executed by a server.
This is a postgresql module allowing to track planning and execution statistics of all
SQL statements executed by a server.
Using this, we can monitor/figure out what the slowest queries executed
on the server are.
Using this, we can monitor/figure out what the slowest queries executed on the server
are.
Resources:
* Postgresql doc: https://www.postgresql.org/docs/13/pgstatstatements.html
* How to enable it: https://www.virtual-dba.com/postgresql-performance-enabling-pg-stat-statements/
* How to use it: https://www.virtual-dba.com/postgresql-performance-identifying-hot-and-slow-queries/
- Postgresql doc: https://www.postgresql.org/docs/13/pgstatstatements.html
- How to enable it:
https://www.virtual-dba.com/postgresql-performance-enabling-pg-stat-statements/
- How to use it:
https://www.virtual-dba.com/postgresql-performance-identifying-hot-and-slow-queries/

99
docs/datanommer_datagrepper/pg_timescaledb.rst
View file

@ -2,61 +2,64 @@ Using the timescaledb extension
===============================
timescaledb (https://docs.timescale.com/latest/) is a postgresql extension for
time-series database.
Considering a lot of the actions done on datagrepper involve the timestamp field
(for example: all the messages with that topic in this time range), we figured
this extension is worth investigating.
A bonus point being for this extension to already packaged and available in
Fedora and EPEL.
time-series database. Considering a lot of the actions done on datagrepper involve the
timestamp field (for example: all the messages with that topic in this time range), we
figured this extension is worth investigating.
A bonus point being for this extension to already packaged and available in Fedora and
EPEL.
Resources
---------
* Setting up/enabling timescaledb: https://severalnines.com/database-blog/how-enable-timescaledb-existing-postgresql-database
* Migrating an existing database to timescaledb: https://docs.timescale.com/latest/getting-started/migrating-data#same-db
- Setting up/enabling timescaledb:
https://severalnines.com/database-blog/how-enable-timescaledb-existing-postgresql-database
- Migrating an existing database to timescaledb:
https://docs.timescale.com/latest/getting-started/migrating-data#same-db
Installing/enabling/activating
------------------------------
To install the plugin, simply run:
::
.. code-block::
dnf install timescaledb
The edit ``/var/lib/pgsql/data/postgresql.conf`` to tell postgresql to load it:
::
.. code-block::
shared_preload_libraries = 'pg_stat_statements,timescaledb'
timescaledb.max_background_workers=4
It will then need a restart of the entire database server:
::
.. code-block::
systemctl restart postgresql
You can then check if the extension loaded properly:
::
.. code-block::
$ sudo -u postgres psql
SELECT * FROM pg_available_extensions ORDER BY name;
Then, you will need to activate it for your database:
::
.. code-block::
$ sudo -u postgres psql <database_name>
CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;
Finally, you can check that the extension was activated for your database:
::
.. code-block::
$ sudo -u postgres psql <database_name>
\dx
.. _timescaledb_findings:
Findings
@ -66,10 +69,8 @@ Partitioned table
~~~~~~~~~~~~~~~~~
After converting the `messages` table to use timescaledb, we've realized that
timescaledb uses table partitioning as well.
This leads to the same issue with the foreign key constraints that we have seen
in the plain partitioning approach we took.
timescaledb uses table partitioning as well. This leads to the same issue with the
foreign key constraints that we have seen in the plain partitioning approach we took.
Foreign key considerations
~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -78,25 +79,25 @@ For a better understanding on the challenges we've encountered with foreign key
constraints, here is a graphical representation of the datanommer database:
.. image:: ../_static/datanommer_db.jpeg
:target: ../_images/datanommer_db.jpeg
:target: ../_images/datanommer_db.jpeg
So here are the issues we've faced:
* To make the `messages` table a hypertable (ie: activate the timescaledb plugin
on it), the tables need to be empty and the data imported in a second step.
* Once the `messages` table is a hypertable, we cannot add foreign key constraints
from the `user_messages` or `package_messages` tables to it. It is just not
supported in timescaledb (cf https://docs.timescale.com/latest/using-timescaledb/schema-management#constraints )
* We tried creating the foreign key constraints before making the `messages` table
a hypertable and then importing the data in (tweaking the primary key and
foreign keys to include the timestamp, following https://stackoverflow.com/questions/64570143/ )
- To make the `messages` table a hypertable (ie: activate the timescaledb plugin on it),
the tables need to be empty and the data imported in a second step.
- Once the `messages` table is a hypertable, we cannot add foreign key constraints from
the `user_messages` or `package_messages` tables to it. It is just not supported in
timescaledb (cf
https://docs.timescale.com/latest/using-timescaledb/schema-management#constraints )
- We tried creating the foreign key constraints before making the `messages` table a
hypertable and then importing the data in (tweaking the primary key and foreign keys
to include the timestamp, following https://stackoverflow.com/questions/64570143/ )
but that resulted in an error when importing the data.
So we ended up with: Keep the same data structure but to not enforce the foreign
key constaints on `user_messages` and `package_messages` to `messages`. As that
database is mostly about inserts and has no updates or deletes, we don't foresee
much problems with this.
So we ended up with: Keep the same data structure but to not enforce the foreign key
constaints on `user_messages` and `package_messages` to `messages`. As that database is
mostly about inserts and has no updates or deletes, we don't foresee much problems with
this.
Duplicated messages
~~~~~~~~~~~~~~~~~~~
@ -105,31 +106,29 @@ When testing datagrepper and datanommer in our test instance with the timescaled
plugin, we saw a number of duplicated messages showing up in the `/raw` endpoint.
Checking if we could fix this server side, we found out that the previous database
schema had an `UNIQUE` constraint on `msg_id` field. However, with the timescaledb
plugin, that constraint is now on both `msg_id` and `timestamp` fields, meaning
a message can be inserted twice in the database if there is a little delay between
the two inserts.
However, migrating datanommer from fedmsg to fedora-messaging should resolve that
issue client side as rabbitmq will ensure there is only one consumer at a time
handling a message.
plugin, that constraint is now on both `msg_id` and `timestamp` fields, meaning a
message can be inserted twice in the database if there is a little delay between the two
inserts.
However, migrating datanommer from fedmsg to fedora-messaging should resolve that issue
client side as rabbitmq will ensure there is only one consumer at a time handling a
message.
Open questions
--------------
* How will upgrading the postgresql version with the timescaledb plugin look like?
It looks like the timescaledb folks are involved enough in postgresql itself that
we think things will work, but we have not had on-hands experience with it.
- How will upgrading the postgresql version with the timescaledb plugin look like?
It looks like the timescaledb folks are involved enough in postgresql itself that we
think things will work, but we have not had on-hands experience with it.
Patch
-----
Here is the patch that needs to be applied to ``datanommer/models/__init__.py``
to get it working with timescaledb's adjusted postgresql model.
Here is the patch that needs to be applied to ``datanommer/models/__init__.py`` to get
it working with timescaledb's adjusted postgresql model.
::
.. code-block::
diff --git a/datanommer.models/datanommer/models/__init__.py b/datanommer.models/datanommer/models/__init__.py
index ada58fa..7780433 100644

135
docs/datanommer_datagrepper/stats.rst
View file

@ -1,8 +1,8 @@
Lies, Damn lies and Statistics
==============================
In order to compare the performances of datagrepper in the different configuration
we looked at, we wrote a small script that runs 30 requests in 10 parallel threads.
In order to compare the performances of datagrepper in the different configuration we
looked at, we wrote a small script that runs 30 requests in 10 parallel threads.
These requests are:
@ -15,22 +15,17 @@ These requests are:
We have then 4 different environments:
- prod/openshift: this is an openshift deployment of datagrepper hitting the
production database, without any configuration change.
- prod/aws: this is an AWS deployment of datagrepper, hitting its own local
database, with the ``DEFAULT_QUERY_DELTA`` configuration key set to 3 days.
- partition/aws: this is an AWS deployment of datagrepper, hitting its own
local postgresql database where the ``messages`` table is partition by ``id``
with each partition having 10 million records and the ``DEFAULT_QUERY_DELTA``
configuration key set to 3 days.
- timescaledb/aws: this is an AWS deployment of datagrepper, hitting its own
local postgresql database where the ``messages`` table as been partition via
the `timescaledb` plugin and the ``DEFAULT_QUERY_DELTA`` configuration key set
to 3 days.
- prod/openshift: this is an openshift deployment of datagrepper hitting the production
database, without any configuration change.
- prod/aws: this is an AWS deployment of datagrepper, hitting its own local database,
with the ``DEFAULT_QUERY_DELTA`` configuration key set to 3 days.
- partition/aws: this is an AWS deployment of datagrepper, hitting its own local
postgresql database where the ``messages`` table is partition by ``id`` with each
partition having 10 million records and the ``DEFAULT_QUERY_DELTA`` configuration key
set to 3 days.
- timescaledb/aws: this is an AWS deployment of datagrepper, hitting its own local
postgresql database where the ``messages`` table as been partition via the
`timescaledb` plugin and the ``DEFAULT_QUERY_DELTA`` configuration key set to 3 days.
Results
-------
@ -40,82 +35,54 @@ Here are the results for each environment and request.
prod/openshift
~~~~~~~~~~~~~~
+--------------------+------------------+-------------------+------------------+-----------------+
| | Requests per sec | Mean time per Req | Max time per Req | Percent success |
+====================+==================+===================+==================+=================+
| filter_by_topic | 0.32 | NA | 45.857601 | 0.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| plain_raw | 0.32 | NA | 31.955371 | 0.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_category | 0.32 | NA | 31.632514 | 0.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_username | 0.32 | NA | 33.549061 | 0.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_package | 0.32 | NA | 34.531207 | 0.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| get_by_id | 1.57 | 1.575608 | 31.259095 | 86.67% |
+--------------------+------------------+-------------------+------------------+-----------------+
================== ==== ======== ========= ======
================== ==== ======== ========= ======
filter_by_topic 0.32 NA 45.857601 0.00%
plain_raw 0.32 NA 31.955371 0.00%
filter_by_category 0.32 NA 31.632514 0.00%
filter_by_username 0.32 NA 33.549061 0.00%
filter_by_package 0.32 NA 34.531207 0.00%
get_by_id 1.57 1.575608 31.259095 86.67%
================== ==== ======== ========= ======
prod/aws
~~~~~~~~
+--------------------+------------------+-------------------+------------------+-----------------+
| | Requests per sec | Mean time per Req | Max time per Req | Percent success |
+====================+==================+===================+==================+=================+
| filter_by_topic | 7.6 | 1.0068 | 11.2743 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| plain_raw | 9.06 | 0.712975 | 3.323922 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_category | 12.43 | 0.489915 | 1.676223 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_username | 1.49 | 5.83623 | 10.661274 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_package | 0 | 52.69256 | 120.229874 | 1.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| get_by_id | 0.73 | 1.534168 | 60.455334 | 83.33% |
+--------------------+------------------+-------------------+------------------+-----------------+
================== ===== ======== ========== =======
================== ===== ======== ========== =======
filter_by_topic 7.6 1.0068 11.2743 100.00%
plain_raw 9.06 0.712975 3.323922 100.00%
filter_by_category 12.43 0.489915 1.676223 100.00%
filter_by_username 1.49 5.83623 10.661274 100.00%
filter_by_package 0 52.69256 120.229874 1.00%
get_by_id 0.73 1.534168 60.455334 83.33%
================== ===== ======== ========== =======
partition/aws
~~~~~~~~~~~~~
+--------------------+------------------+-------------------+------------------+-----------------+
| | Requests per sec | Mean time per Req | Max time per Req | Percent success |
+====================+==================+===================+==================+=================+
| filter_by_topic | 9.98 | 0.711219 | 3.204178 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| plain_raw | 9.70 | 0.641497 | 1.24704 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_category | 13.32 | 0.455219 | 0.594465 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_username | 1.3 | 7.084018 | 12.079198 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_package | 0 | 55.231556 | 120.125013 | 1.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| get_by_id | 0.48 | 2.198211 | 60.444765 | 76.67% |
+--------------------+------------------+-------------------+------------------+-----------------+
================== ===== ========= ========== =======
================== ===== ========= ========== =======
filter_by_topic 9.98 0.711219 3.204178 100.00%
plain_raw 9.70 0.641497 1.24704 100.00%
filter_by_category 13.32 0.455219 0.594465 100.00%
filter_by_username 1.3 7.084018 12.079198 100.00%
filter_by_package 0 55.231556 120.125013 1.00%
get_by_id 0.48 2.198211 60.444765 76.67%
================== ===== ========= ========== =======
timescaledb/aws
~~~~~~~~~~~~~~~
+--------------------+------------------+-------------------+------------------+-----------------+
| | Requests per sec | Mean time per Req | Max time per Req | Percent success |
+====================+==================+===================+==================+=================+
| filter_by_topic | 14.1 | 0.4286 | 0.514617 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| plain_raw | 12.89 | 0.48235 | 0.661073 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_category | 13.94 | 0.423172 | 0.507337 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_username | 2.68 | 3.188782 | 5.096244 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| filter_by_package | 0.26 | 33.216631 | 57.901159 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
| get_by_id | 12.69 | 0.749068 | 1.73515 | 100.00% |
+--------------------+------------------+-------------------+------------------+-----------------+
================== ===== ========= ========= =======
================== ===== ========= ========= =======
filter_by_topic 14.1 0.4286 0.514617 100.00%
plain_raw 12.89 0.48235 0.661073 100.00%
filter_by_category 13.94 0.423172 0.507337 100.00%
filter_by_username 2.68 3.188782 5.096244 100.00%
filter_by_package 0.26 33.216631 57.901159 100.00%
get_by_id 12.69 0.749068 1.73515 100.00%
================== ===== ========= ========= =======
Graphs
------
@ -128,24 +95,20 @@ Percentage of success
.. image:: ../_static/datanommer_percent_sucess.jpg
:target: ../_images/datanommer_percent_sucess.jpg
Requests per second
~~~~~~~~~~~~~~~~~~~
.. image:: ../_static/datanommer_req_per_sec.jpg
:target: ../_images/datanommer_req_per_sec.jpg
Mean time per request
~~~~~~~~~~~~~~~~~~~~~
.. image:: ../_static/datanommer_mean_per_req.jpg
:target: ../_images/datanommer_mean_per_req.jpg
Maximum time per request
~~~~~~~~~~~~~~~~~~~~~~~~
.. image:: ../_static/datanommer_max_per_req.jpg
:target: ../_images/datanommer_max_per_req.jpg

46
docs/dist-git-move/bodhi.rst
View file

@ -1,43 +1,45 @@
.. _bodhi:
Pagure Dist Git Interactions With Bodhi
====
=======================================
`Bodhi <https://bodhi.fedoraproject.org>`_ is a service
that handles package updates in Fedora Linux. It works as a gating system for
Fedora updates and allows users to test the updates before being published in
distribution.
`Bodhi <https://bodhi.fedoraproject.org>`__ is a service that handles package updates in
Fedora Linux. It works as a gating system for Fedora updates and allows users to test
the updates before being published in distribution.
Possible Interactions
----
---------------------
The following are a set of documented interactions possible between Pagure Dist
Git and Bodhi.
The following are a set of documented interactions possible between Pagure Dist Git and
Bodhi.
Configuration
^^^^
~~~~~~~~~~~~~
- The configuration in Bodhi uses ``pagure_url`` for dist git, default is
defined in
`config.py <https://github.com/fedora-infra/bodhi/blob/develop/bodhi-server/bodhi/server/config.py>`_
- The configuration in Bodhi uses ``pagure_url`` for dist git, default is defined in
`config.py
<https://github.com/fedora-infra/bodhi/blob/develop/bodhi-server/bodhi/server/config.py>`_
API
^^^^
~~~
- Bodhi calls following API endpoints:
* ``/api/0/<namespace>/<package_name>`` to get information about contributors for the package
* ``/api/0/<namespace>/<package_name>/hascommit`` to check if user has commit access to package
- ``/api/0/<namespace>/<package_name>`` to get information about contributors for the
package
- ``/api/0/<namespace>/<package_name>/hascommit`` to check if user has commit access
to package
Changes proposed
----
----------------
In order to switch out Pagure Dist Git for an alternative that provides a
relatively compatible API pattern, the following changes must be made in the
places mentioned below.
In order to switch out Pagure Dist Git for an alternative that provides a relatively
compatible API pattern, the following changes must be made in the places mentioned
below.
Necessary
^^^^
~~~~~~~~~
- If URL for dist git is change it will need to be reflected in
`deployment configuration <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
- If URL for dist git is change it will need to be reflected in `deployment
configuration
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_

67
docs/dist-git-move/ci.rst
View file

@ -1,69 +1,68 @@
.. _ci:
Pagure Dist Git Interactions With Fedora CI
====
===========================================
`Fedora CI <https://docs.fedoraproject.org/en-US/ci/>`_ is a service
that handles Continuous Integration of packages in Fedora project.
As that it interacts with dist-git.
Fedora CI consists of multiple services:
`Fedora CI <https://docs.fedoraproject.org/en-US/ci/>`_ is a service that handles
Continuous Integration of packages in Fedora project. As that it interacts with
dist-git. Fedora CI consists of multiple services:
- Jenkins - Fedora hosted instance that runs the tests
- `Greenwave <https://pagure.io/greenwave>`_ - service to evaluate
gating policies on test results
- `ResultsDB <https://pagure.io/taskotron/resultsdb>`_ - stores
results of the tests
- `WaiverDB <https://pagure.io/waiverdb>`_ - service for recording
waivers against test results
- `Greenwave <https://pagure.io/greenwave>`__ - service to evaluate gating policies on
test results
- `ResultsDB <https://pagure.io/taskotron/resultsdb>`_ - stores results of the tests
- `WaiverDB <https://pagure.io/waiverdb>`_ - service for recording waivers against test
results
Possible Interactions
----
---------------------
The following are a set of documented interactions possible between Pagure Dist
Git and Fedora CI.
The following are a set of documented interactions possible between Pagure Dist Git and
Fedora CI.
Rawhide gating
^^^^
~~~~~~~~~~~~~~
- To configure rawhide gating rules for Greenwave user needs to add ``gating.yaml`` file
to package repository
Notification about results
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~
- Notification about CI results is done by creating flags using API calls:
* ``/api/0/<package_name>/pull-request/<pr_id>/flag``
* ``/api/0/<package_name>/c/<commit_id>/flag``
- ``/api/0/<package_name>/pull-request/<pr_id>/flag``
- ``/api/0/<package_name>/c/<commit_id>/flag``
Those flags are then visible on the corresponding commit or pull request.
Triggering tests on pull requests
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The automatic trigger of CI when creating pull request is done by emitting
message topic ``pull-request.new`` and it's consumed by Jenkins
- The automatic trigger of CI when creating pull request is done by emitting message
topic ``pull-request.new`` and it's consumed by Jenkins
Changes proposed
----
----------------
In order to switch out Pagure Dist Git for an alternative that provides a
relatively compatible API pattern, the following changes must be made in the
places mentioned below.
In order to switch out Pagure Dist Git for an alternative that provides a relatively
compatible API pattern, the following changes must be made in the places mentioned
below.
Necessary
^^^^
~~~~~~~~~
- If URL for dist git is change it will need to be reflected in each component
that is part of Fedora CI and directly interacts with dist-git:
- If URL for dist git is change it will need to be reflected in each component that is
part of Fedora CI and directly interacts with dist-git:
* `Greenwave <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/greenwave/templates/settings.py>`_ -
HTTP get request to the repository to check the ``gating.yaml``
- `Greenwave
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/greenwave/templates/settings.py>`__
- HTTP get request to the repository to check the ``gating.yaml``
Unnecessary
^^^^
~~~~~~~~~~~
- In case message schema will change we would need to update the topics that
are consumed by Fedora CI. The
`configuration <https://github.com/fedora-ci/dist-git-build-trigger/blob/pull-request.new/Jenkinsfile>`_
- In case message schema will change we would need to update the topics that are
consumed by Fedora CI. The `configuration
<https://github.com/fedora-ci/dist-git-build-trigger/blob/pull-request.new/Jenkinsfile>`_
for Jenkins is here.

70
docs/dist-git-move/copr.rst
View file

@ -1,66 +1,68 @@
.. _copr:
Pagure Dist Git Interactions With COPR
====
======================================
`Copr <https://copr.fedorainfracloud.org/>`_ ("Community projects") is a service
that builds your open-source projects and creates your own RPM repositories.
`Copr <https://copr.fedorainfracloud.org/>`__ ("Community projects") is a service that
builds your open-source projects and creates your own RPM repositories.
Possible Interactions
----
---------------------
The following are a set of documented interactions possible between Pagure Dist
Git and COPR.
The following are a set of documented interactions possible between Pagure Dist Git and
COPR.
Builds logic
^^^^
~~~~~~~~~~~~
- COPR is using API calls for flagging dist-git commits or pull requests in
`build_logic.py module <https://github.com/fedora-copr/copr/blob/2789f06b36be5c13eae83c8695eb36de03a7728f/frontend/coprs_frontend/coprs/logic/builds_logic.py#L1101-L1109>`_.
`build_logic.py module
<https://github.com/fedora-copr/copr/blob/2789f06b36be5c13eae83c8695eb36de03a7728f/frontend/coprs_frontend/coprs/logic/builds_logic.py#L1101-L1109>`_.
It is calling ``/api/0/<package_name>/pull-request/<pr_id>/flag`` and
``/api/0/<package_name>/c/<commit_id>/flag`` API endpoints.
CLI client
^^^^
~~~~~~~~~~
- COPR is doing ``git clone`` from dist-git, which is wrapped in
``copr-distgit-client clone --dist-git fedora <pkgname>``.
- COPR is doing ``git clone`` from dist-git, which is wrapped in ``copr-distgit-client
clone --dist-git fedora <pkgname>``.
Fedora messaging
^^^^
~~~~~~~~~~~~~~~~
- COPR is listening to following dist-git topics:
* ``org.fedoraproject.prod.pagure.git.receive``
* ``org.fedoraproject.prod.pagure.pull-request.new``
* ``org.fedoraproject.prod.pagure.pull-request.rebased``
* ``org.fedoraproject.prod.pagure.pull-request.updated``
* ``org.fedoraproject.prod.pagure.pull-request.comment.added``
- ``org.fedoraproject.prod.pagure.git.receive``
- ``org.fedoraproject.prod.pagure.pull-request.new``
- ``org.fedoraproject.prod.pagure.pull-request.rebased``
- ``org.fedoraproject.prod.pagure.pull-request.updated``
- ``org.fedoraproject.prod.pagure.pull-request.comment.added``
- These topics are configured in
`frontend messaging configuration <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/copr/frontend/templates/fedora-messaging/copr_messaging.toml>`_.
- These topics are configured in `frontend messaging configuration
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/copr/frontend/templates/fedora-messaging/copr_messaging.toml>`_.
Changes proposed
----
----------------
In order to switch out Pagure Dist Git for an alternative that provides a
relatively compatible API pattern, the following changes must be made in the
places mentioned below.
In order to switch out Pagure Dist Git for an alternative that provides a relatively
compatible API pattern, the following changes must be made in the places mentioned
below.
Necessary
^^^^
~~~~~~~~~
- There are few configuration files that points to current dist-git. If the
URL should change we would need to update
`rpkg.conf <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/copr/backend/templates/provision/rpkg.conf.j2>`_
and
`main.ini <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/copr/backend/templates/provision/copr-rpmbuild/main.ini.j2>`_
- There are few configuration files that points to current dist-git. If the URL should
change we would need to update `rpkg.conf
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/copr/backend/templates/provision/rpkg.conf.j2>`_
and `main.ini
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/copr/backend/templates/provision/copr-rpmbuild/main.ini.j2>`_
with the URL to new service.
- In case the message schema of the new dist-git will change we need to update consumed topics in
`frontend messaging configuration <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/copr/frontend/templates/fedora-messaging/copr_messaging.toml>`_.
- In case the message schema of the new dist-git will change we need to update consumed
topics in `frontend messaging configuration
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/copr/frontend/templates/fedora-messaging/copr_messaging.toml>`_.
Unnecessary
^^^^
~~~~~~~~~~~
- There isn't much interactions between Fedora dist-git and COPR and no change
is needed that would be unnecessary to do in case dist-git will change.
- There isn't much interactions between Fedora dist-git and COPR and no change is needed
that would be unnecessary to do in case dist-git will change.

56
docs/dist-git-move/fedpkg.rst
View file

@ -1,55 +1,55 @@
.. _fedpkg:
Pagure Dist Git Interactions With fedpkg
====
========================================
`Fedpkg <https://pagure.io/fedpkg>`_ is a CLI tool for packagers that allows
them to work with packages in Fedora package workflow.
`Fedpkg <https://pagure.io/fedpkg>`__ is a CLI tool for packagers that allows them to
work with packages in Fedora package workflow.
Possible Interactions
----
---------------------
The following are a set of documented interactions possible between Pagure Dist
Git and fedpkg.
The following are a set of documented interactions possible between Pagure Dist Git and
fedpkg.
Lookaside cache
^^^^
~~~~~~~~~~~~~~~
- fedpkg is using lookaside cache that is located on same server as dist git
https://src.fedoraproject.org/repo/pkgs
API calls
^^^^
~~~~~~~~~
- Fedpkg is calling dist git API endpoint ``api/0/fork`` when creating fork of
the project.
- Fedpkg is calling dist git API endpoint ``_db/anitya/<namespace>/<package_name>``
for disabling monitoring for repository.
- Fedpkg is calling dist git API endpoint ``api/0/fork`` when creating fork of the
project.
- Fedpkg is calling dist git API endpoint ``_db/anitya/<namespace>/<package_name>`` for
disabling monitoring for repository.
Git interactions
^^^^
~~~~~~~~~~~~~~~~
- Fedpkg works with cloned dist git repositories using standard
git commands.
- Fedpkg works with cloned dist git repositories using standard git commands.
Changes proposed
----
----------------
In order to switch out Pagure Dist Git for an alternative that provides a
relatively compatible API pattern, the following changes must be made in the
places mentioned below.
In order to switch out Pagure Dist Git for an alternative that provides a relatively
compatible API pattern, the following changes must be made in the places mentioned
below.
Necessary
^^^^
~~~~~~~~~
- In case URL will be changed default configuration distributed with
fedpkg RPM package needs to be updated.
- In case URL will be changed default configuration distributed with fedpkg RPM package
needs to be updated.
* `fedpkg.conf <https://pagure.io/fedpkg/blob/master/f/conf/etc/rpkg/fedpkg.conf>`_
- `fedpkg.conf <https://pagure.io/fedpkg/blob/master/f/conf/etc/rpkg/fedpkg.conf>`_
for production fedpkg
* `fedpkg-stage.conf <https://pagure.io/fedpkg/blob/master/f/conf/etc/rpkg/fedpkg-stage.conf>`_
for staging fedpkg
- `fedpkg-stage.conf
<https://pagure.io/fedpkg/blob/master/f/conf/etc/rpkg/fedpkg-stage.conf>`_ for
staging fedpkg
- The lookaside cache needs to be still available. In case the location will be different
some changes will be needed with working the lookaside cache in fedpkg as currently it
expects the URL is same as for dist git.
- The lookaside cache needs to be still available. In case the location will be
different some changes will be needed with working the lookaside cache in fedpkg as
currently it expects the URL is same as for dist git.

54
docs/dist-git-move/hotness.rst
View file

@ -1,49 +1,51 @@
.. _hotness:
Pagure Dist Git Interactions With The New Hotness
====
=================================================
Possible Interactions
----
---------------------
The following are a set of documented interactions possible between Pagure Dist
Git and The New Hotness.
The following are a set of documented interactions possible between Pagure Dist Git and
The New Hotness.
Validations
^^^^
~~~~~~~~~~~
- Checking the monitoring setting by calling dist-git ``_dg/anitya/rpms/<package_name>`` API in
`pagure module <https://github.com/fedora-infra/the-new-hotness/blob/master/hotness/validators/pagure.py>`_
- Checking the retirement status by HTTP GET request for ``dead.package`` in
`pagure module <https://github.com/fedora-infra/the-new-hotness/blob/master/hotness/validators/pagure.py>`_
- Checking the monitoring setting by calling dist-git ``_dg/anitya/rpms/<package_name>``
API in `pagure module
<https://github.com/fedora-infra/the-new-hotness/blob/master/hotness/validators/pagure.py>`_
- Checking the retirement status by HTTP GET request for ``dead.package`` in `pagure
module
<https://github.com/fedora-infra/the-new-hotness/blob/master/hotness/validators/pagure.py>`_
Scratch builds
^^^^
~~~~~~~~~~~~~~
- hotness is doing the scratch builds if requested by users in
`koji module <https://github.com/fedora-infra/the-new-hotness/blob/master/hotness/builders/koji.py#L119>`_.
- hotness is doing the scratch builds if requested by users in `koji module
<https://github.com/fedora-infra/the-new-hotness/blob/master/hotness/builders/koji.py#L119>`_.
It uses standard git commands (clone, commit) when preparing the scratch build
Changes proposed
----
----------------
In order to switch out Pagure Dist Git for an alternative that provides a
relatively compatible API pattern, the following changes must be made in the
places mentioned below.
In order to switch out Pagure Dist Git for an alternative that provides a relatively
compatible API pattern, the following changes must be made in the places mentioned
below.
Necessary
^^^^
~~~~~~~~~
- Change the
`configuration <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/the-new-hotness/templates/config.toml>`_
- Change the `configuration
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/the-new-hotness/templates/config.toml>`_
to point to new dist-git instance.
Unnecessary
^^^^
~~~~~~~~~~~
- As hotness is mostly using just plain HTTP, git interactions with dist-git it's not necessary
to change much. Only the monitoring settings needs to be provided somewhere.
- Change the default value for ``dist_git_url`` in
`config module <https://github.com/fedora-infra/the-new-hotness/blob/master/hotness/config.py>`_.
As this could be set in configuration it's not necessary to do that, but it would be nice to
change it to new git forge.
- As hotness is mostly using just plain HTTP, git interactions with dist-git it's not
necessary to change much. Only the monitoring settings needs to be provided somewhere.
- Change the default value for ``dist_git_url`` in `config module
<https://github.com/fedora-infra/the-new-hotness/blob/master/hotness/config.py>`_. As
this could be set in configuration it's not necessary to do that, but it would be nice
to change it to new git forge.

101
docs/dist-git-move/index.rst
View file

@ -1,44 +1,48 @@
Dist Git Move
====
=============
Purpose
----
-------
The objective of the potential initiative is to move repository contents (including but not
limited to source codes, Packit configurations, RPM specfiles) from Pagure Dist-Git to
another platform and confirm that the associated tooling and services (including but not limited
to FMN, Datanommer, COPR, Toddlers, FMN, CI, Monitor-Gating, Packit, Bodhi, Fedpkg) work well with
the newer platform. The investigation aims to be as agnostic as it can be regarding the
destination platform to help ideate a general solution for the compatibility of the associated
tooling and services.
The objective of the potential initiative is to move repository contents (including but
not limited to source codes, Packit configurations, RPM specfiles) from Pagure Dist-Git
to another platform and confirm that the associated tooling and services (including but
not limited to FMN, Datanommer, COPR, Toddlers, FMN, CI, Monitor-Gating, Packit, Bodhi,
Fedpkg) work well with the newer platform. The investigation aims to be as agnostic as
it can be regarding the destination platform to help ideate a general solution for the
compatibility of the associated tooling and services.
Background
----
----------
With Pagure's development having less activity for a while now, we need to ensure that the
workflow changes are documented associated tooling and services are changed to continue working
the way they did when Pagure Dist-Git is decomissioned and the repository contents are moved
elsewhere, while confirming that we are not locked down to the features that are specific to the
chosen destination platform.
With Pagure's development having less activity for a while now, we need to ensure that
the workflow changes are documented associated tooling and services are changed to
continue working the way they did when Pagure Dist-Git is decomissioned and the
repository contents are moved elsewhere, while confirming that we are not locked down to
the features that are specific to the chosen destination platform.
Requirements
----
------------
* Study the interactions of toolings/services with Dist-Git
* Include additions to **Datanommer** to support interactions with another platform
* Include additions to **Fedora Notifications** to support interactions with another platform
* Include additions to **COPR** to support interactions with another platform
* Include additions to **Toddlers** to support interactions with another platform
* Include additions to **Continuous Integration** to support interactions with another platform
* Include additions to **Release Engineering** to support interactions with another platform
* Include additions to **Monitor Gating** to support interactions with another platform
* Include additions to **Packit** to support interactions with another platform
* Include additions to **Bodhi** to support interactions with another platform
* Include additions to **Pagure** to support interactions with another platform
* Include additions to **Anitya/The New Hotness** to support interactions with another platform
- Study the interactions of toolings/services with Dist-Git
- Include additions to **Datanommer** to support interactions with another platform
- Include additions to **Fedora Notifications** to support interactions with another
platform
- Include additions to **COPR** to support interactions with another platform
- Include additions to **Toddlers** to support interactions with another platform
- Include additions to **Continuous Integration** to support interactions with another
platform
- Include additions to **Release Engineering** to support interactions with another
platform
- Include additions to **Monitor Gating** to support interactions with another platform
- Include additions to **Packit** to support interactions with another platform
- Include additions to **Bodhi** to support interactions with another platform
- Include additions to **Pagure** to support interactions with another platform
- Include additions to **Anitya/The New Hotness** to support interactions with another
platform
Index
----
-----
.. toctree::
:maxdepth: 1
@ -56,32 +60,33 @@ Index
releng
Conclusions
----
-----------
We are concluding that we would be filling this section with a placeholder for now. The best
one in the placeholder business is called Lorem Ipsum. Lorem Ipsum is simply dummy text of the
printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever
since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type
specimen book. It has survived not only five centuries, but also the leap into electronic
typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release
of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing
software like Aldus PageMaker including versions of Lorem Ipsum.
We are concluding that we would be filling this section with a placeholder for now. The
best one in the placeholder business is called Lorem Ipsum. Lorem Ipsum is simply dummy
text of the printing and typesetting industry. Lorem Ipsum has been the industry's
standard dummy text ever since the 1500s, when an unknown printer took a galley of type
and scrambled it to make a type specimen book. It has survived not only five centuries,
but also the leap into electronic typesetting, remaining essentially unchanged. It was
popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum
passages, and more recently with desktop publishing software like Aldus PageMaker
including versions of Lorem Ipsum.
Roadmap
----
-------
- **Step 1** - Start doing something
- **Step 2** - Continue doing something
- **Step 3** - Stop doing something
Efforts
---
-------
We are estimating nothing at the moment so here is some more of that Lorem Ipsum text that you
all know and love. Lorem Ipsum is simply dummy text of the printing and typesetting industry.
Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown
printer took a galley of type and scrambled it to make a type specimen book. It has survived not
only five centuries, but also the leap into electronic typesetting, remaining essentially
unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem
Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including
versions of Lorem Ipsum.
We are estimating nothing at the moment so here is some more of that Lorem Ipsum text
that you all know and love. Lorem Ipsum is simply dummy text of the printing and
typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since
the 1500s, when an unknown printer took a galley of type and scrambled it to make a type
specimen book. It has survived not only five centuries, but also the leap into
electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s
with the release of Letraset sheets containing Lorem Ipsum passages, and more recently
with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.

584
docs/dist-git-move/messaging.rst
View file

@ -1,478 +1,466 @@
.. _messaging:
Pagure Dist Git Interactions With Fedora Messaging
====
==================================================
Fedora Messaging provides tools and APIs to make using Fedora Project's
messaging infrastructure easier. These include a framework for declaring
message schemas, a set of asynchronous APIs to consume messages, and services
to easily run consumers.
Fedora Messaging provides tools and APIs to make using Fedora Project's messaging
infrastructure easier. These include a framework for declaring message schemas, a set of
asynchronous APIs to consume messages, and services to easily run consumers.
Interaction Dependencies
----
------------------------
Fedora Messaging makes use of the following dependencies to interact with and
understand interactions made with Pagure Dist Git.
Fedora Messaging makes use of the following dependencies to interact with and understand
interactions made with Pagure Dist Git.
Pagure Messages
^^^^
~~~~~~~~~~~~~~~
In order for `Fedora Messaging <https://fedora-messaging.readthedocs.io/en/stable/>`_
to comprehend the messages emitted from Pagure, it requires a helper library
called `Pagure Messages <https://pagure.io/pagure-messages>`_. This package is
distributed in the official Fedora Project repositories
`here <https://src.fedoraproject.org/rpms/python-pagure-messages>`_ and on PyPI
`here <https://pypi.org/project/pagure-messages/>`_. It is important to note
that at the time of writing this investigation, the former seems to be in an
unmaintained state and the last update i.e. ``v0.0.6`` was created around two
years back. The PyPI releases are fairly up-to-date with the most recent
release i.e. ``v1.1.0`` being created around four months back.
In order for `Fedora Messaging <https://fedora-messaging.readthedocs.io/en/stable/>`_ to
comprehend the messages emitted from Pagure, it requires a helper library called `Pagure
Messages <https://pagure.io/pagure-messages>`_. This package is distributed in the
official Fedora Project repositories `here
<https://src.fedoraproject.org/rpms/python-pagure-messages>`__. and on PyPI `here
<https://pypi.org/project/pagure-messages/>`__. It is important to note that at the time
of writing this investigation, the former seems to be in an unmaintained state and the
last update i.e. ``v0.0.6`` was created around two years back. The PyPI releases are
fairly up-to-date with the most recent release i.e. ``v1.1.0`` being created around four
months back.
Possible Interactions
----
---------------------
The following are a set of documented interactions possible between Pagure Dist
Git and Fedora Messaging.
The following are a set of documented interactions possible between Pagure Dist Git and
Fedora Messaging.
Projects
^^^^
~~~~~~~~
The messages in this category follow the topic name pattern ``pagure.project.*``.
Creation of a project
""""
+++++++++++++++++++++
Whenever a new project is created on either Pagure or Pagure Dist Git, the
message of schema topic ``pagure.project.new`` is created on the Fedora
Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_20-57>`_.
Whenever a new project is created on either Pagure or Pagure Dist Git, the message of
schema topic ``pagure.project.new`` is created on the Fedora Messaging bus. The class
definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_20-57>`__.
Editing of a project
""""
++++++++++++++++++++
Whenever an existing project is edited on either Pagure or Pagure Dist Git, the
message of schema topic ``pagure.project.edit`` is created on the Fedora
Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_60-99>`_.
Whenever an existing project is edited on either Pagure or Pagure Dist Git, the message
of schema topic ``pagure.project.edit`` is created on the Fedora Messaging bus. The
class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_60-99>`__.
Forking of a project
""""
++++++++++++++++++++
Whenever an existing project is forked on either Pagure or Pagure Dist Git, the
message of schema topic ``pagure.project.forked`` is created on the Fedora
Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_102-140>`_.
Whenever an existing project is forked on either Pagure or Pagure Dist Git, the message
of schema topic ``pagure.project.forked`` is created on the Fedora Messaging bus. The
class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_102-140>`__.
Deleting of a project
""""
+++++++++++++++++++++
Whenever an existing project is removed from either Pagure or Pagure Dist Git,
the message of schema topic ``pagure.project.deleted`` is created on the Fedora
Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_143-180>`_.
Whenever an existing project is removed from either Pagure or Pagure Dist Git, the
message of schema topic ``pagure.project.deleted`` is created on the Fedora Messaging
bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_143-180>`__.
Adding a group to a project
""""
+++++++++++++++++++++++++++
Whenever access is provided to a group for a project on either Pagure or Pagure
Dist Git, the message of schema topic ``pagure.project.group.added`` is created
on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_183-229>`_.
Whenever access is provided to a group for a project on either Pagure or Pagure Dist
Git, the message of schema topic ``pagure.project.group.added`` is created on the Fedora
Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_183-229>`__.
Deleting a group from a project
""""
+++++++++++++++++++++++++++++++
Whenever access is revoked from a group for a project on either Pagure or
Pagure Dist Git, the message of schema topic ``pagure.project.group.removed``
is created on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_232-279>`_.
Whenever access is revoked from a group for a project on either Pagure or Pagure Dist
Git, the message of schema topic ``pagure.project.group.removed`` is created on the
Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_232-279>`__.
Updating a group on a project
""""
+++++++++++++++++++++++++++++
Whenever access is updated to a group for a project on either Pagure or Pagure
Dist Git, the message of schema topic ``pagure.project.group.access.updated``
is created on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_282-326>`_.
Whenever access is updated to a group for a project on either Pagure or Pagure Dist Git,
the message of schema topic ``pagure.project.group.access.updated`` is created on the
Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_282-326>`__.
Editing a tag on a project
""""
++++++++++++++++++++++++++
Whenever tags are edited for a project on either Pagure or Pagure Dist Git, the
message of schema topic ``pagure.project.tag.edited`` is created on the Fedora
Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_329-368>`_.
Whenever tags are edited for a project on either Pagure or Pagure Dist Git, the message
of schema topic ``pagure.project.tag.edited`` is created on the Fedora Messaging bus.
The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_329-368>`__.
Deleting a tag on a project
""""
+++++++++++++++++++++++++++
Whenever tags are deleted from a project on either Pagure or Pagure Dist Git,
the message of schema topic ``pagure.project.tag.removed`` is created on the
Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_371-410>`_.
Whenever tags are deleted from a project on either Pagure or Pagure Dist Git, the
message of schema topic ``pagure.project.tag.removed`` is created on the Fedora
Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_371-410>`__.
Updating a user to a project
""""
++++++++++++++++++++++++++++
Whenever access is updated to a user for a project on either Pagure or Pagure
Dist Git, the message of schema topic ``pagure.project.user.access.updated`` is
created on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_413-457>`_.
Whenever access is updated to a user for a project on either Pagure or Pagure Dist Git,
the message of schema topic ``pagure.project.user.access.updated`` is created on the
Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_413-457>`__.
Adding a user to a project
""""
++++++++++++++++++++++++++
Whenever access is provided to a user for a project on either Pagure or Pagure
Dist Git, the message of schema topic ``pagure.project.user.added`` is created
on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_460-499>`_.
Whenever access is provided to a user for a project on either Pagure or Pagure Dist Git,
the message of schema topic ``pagure.project.user.added`` is created on the Fedora
Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_460-499>`__.
Deleting a user from a project
""""
++++++++++++++++++++++++++++++
Whenever access is revoked from a user for a project on either Pagure or Pagure
Dist Git, the message of schema topic ``pagure.project.user.removed`` is
created on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_502-541>`_.
Whenever access is revoked from a user for a project on either Pagure or Pagure Dist
Git, the message of schema topic ``pagure.project.user.removed`` is created on the
Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/project_schema.py#_502-541>`__.
Repositories
^^^^
~~~~~~~~~~~~
The messages in this category follow the topic name pattern ``pagure.git.*``.
Creation of a branch
""""
++++++++++++++++++++
Whenever a new branch is created on a project on either Pagure or Pagure Dist
Git, the message of schema topic ``pagure.git.branch.creation`` is created on
the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/git_schema.py#_47-95>`_.
Whenever a new branch is created on a project on either Pagure or Pagure Dist Git, the
message of schema topic ``pagure.git.branch.creation`` is created on the Fedora
Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/git_schema.py#_47-95>`__.
Deletion of a branch
""""
++++++++++++++++++++
Whenever an existing branch is deleted on a project on either Pagure or Pagure
Dist Git, the message of schema topic ``pagure.git.branch.deletion`` is created
on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/git_schema.py#_98-139>`_.
Whenever an existing branch is deleted on a project on either Pagure or Pagure Dist Git,
the message of schema topic ``pagure.git.branch.deletion`` is created on the Fedora
Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/git_schema.py#_98-139>`__.
Reception of commits
""""
++++++++++++++++++++
Whenever a set of commits are pushed to an existing branch in a repository on
either Pagure or Pagure Dist Git, the message of schema topic
``pagure.git.receive`` is created on the Fedora Messaging bus. The class
definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/git_schema.py#_142-224>`_.
Whenever a set of commits are pushed to an existing branch in a repository on either
Pagure or Pagure Dist Git, the message of schema topic ``pagure.git.receive`` is created
on the Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/git_schema.py#_142-224>`__.
Creation of a tag
""""
+++++++++++++++++
Whenever a new tag is created on a project on either Pagure or Pagure Dist Git,
the message of schema topic ``pagure.git.tag.creation`` is created on the
Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/git_schema.py#_227-273>`_.
Whenever a new tag is created on a project on either Pagure or Pagure Dist Git, the
message of schema topic ``pagure.git.tag.creation`` is created on the Fedora Messaging
bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/git_schema.py#_227-273>`__.
Deletion of a tag
""""
+++++++++++++++++
Whenever an existing tag is deleted on a project on either Pagure or Pagure
Dist Git, the message of schema topic ``pagure.git.tag.deletion`` is created
on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/git_schema.py#_276-320>`_.
Whenever an existing tag is deleted on a project on either Pagure or Pagure Dist Git,
the message of schema topic ``pagure.git.tag.deletion`` is created on the Fedora
Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/git_schema.py#_276-320>`__.
Issues
^^^^
~~~~~~
The messages in this category follow the topic name pattern ``pagure.issue.*``.
Please note that these are not used in Pagure Dist Git as the **Issues** tab in
Pagure Dist Git points to the Bugzilla page related to the said package.
Please note that these are not used in Pagure Dist Git as the **Issues** tab in Pagure
Dist Git points to the Bugzilla page related to the said package.
Adding assignment to an issue ticket
""""
++++++++++++++++++++++++++++++++++++
Whenever an assignment is added to an existing issue ticket on a project
present on Pagure, the message of schema topic ``pagure.issue.assigned.added``
is created on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_20-62>`_.
Whenever an assignment is added to an existing issue ticket on a project present on
Pagure, the message of schema topic ``pagure.issue.assigned.added`` is created on the
Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_20-62>`__.
Removing assignment from an issue ticket
""""
++++++++++++++++++++++++++++++++++++++++
Whenever an assignment is removed from an existing issue ticket on a project
present on Pagure, the message of schema topic
``pagure.issue.assigned.reset`` is created on the Fedora Messaging bus. The
class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_65-105>`_.
Whenever an assignment is removed from an existing issue ticket on a project present on
Pagure, the message of schema topic ``pagure.issue.assigned.reset`` is created on the
Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_65-105>`__.
Comment added to an issue ticket
""""
++++++++++++++++++++++++++++++++
Whenever a comment is added to an existing issue ticket on a project present on
either Pagure and Pagure Dist Git, the message of schema topic
``pagure.issue.comment.added`` is created on the Fedora Messaging bus. The
class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_108-153>`_.
Whenever a comment is added to an existing issue ticket on a project present on either
Pagure and Pagure Dist Git, the message of schema topic ``pagure.issue.comment.added``
is created on the Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_108-153>`__.
Adding dependency to an issue ticket
""""
++++++++++++++++++++++++++++++++++++
Whenever a dependency is added to an existing issue ticket on a project present
on Pagure, the message of schema topic ``pagure.issue.dependency.added`` is
created on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_156-201>`_.
Whenever a dependency is added to an existing issue ticket on a project present on
Pagure, the message of schema topic ``pagure.issue.dependency.added`` is created on the
Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_156-201>`__.
Removing dependency from an issue ticket
""""
++++++++++++++++++++++++++++++++++++++++
Whenever a dependency is removed from an existing issue ticket from a project
present on Pagure, the message of schema topic
``pagure.issue.dependency.removed`` is created on the Fedora Messaging bus. The
class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_204-257>`_.
Whenever a dependency is removed from an existing issue ticket from a project present on
Pagure, the message of schema topic ``pagure.issue.dependency.removed`` is created on
the Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_204-257>`__.
Removing an issue ticket
""""
++++++++++++++++++++++++
Whenever an existing issue ticket is removed from a project present on either
Pagure, the message of schema topic ``pagure.issue.drop`` is created on the
Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_260-303>`_.
Whenever an existing issue ticket is removed from a project present on either Pagure,
the message of schema topic ``pagure.issue.drop`` is created on the Fedora Messaging
bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_260-303>`__.
Editing an issue ticket
""""
+++++++++++++++++++++++
Whenever an existing issue ticket is edited on a project present on either
Pagure, the message of schema topic ``pagure.issue.edit`` is created on the
Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_306-351>`_.
Whenever an existing issue ticket is edited on a project present on either Pagure, the
message of schema topic ``pagure.issue.edit`` is created on the Fedora Messaging bus.
The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_306-351>`__.
Creating an issue ticket
""""
++++++++++++++++++++++++
Whenever a new issue ticket is created on a project present on either Pagure or
, the message of schema topic ``pagure.issue.new`` is created on the Fedora
Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_354-395>`_.
Whenever a new issue ticket is created on a project present on either Pagure or , the
message of schema topic ``pagure.issue.new`` is created on the Fedora Messaging bus. The
class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_354-395>`__.
Adding label to an issue ticket
""""
+++++++++++++++++++++++++++++++
Whenever labels are added to an existing issue ticket is created on a project
present on Pagure, the message of schema topic ``pagure.issue.tag.added`` is
created on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_398-441>`_.
Whenever labels are added to an existing issue ticket is created on a project present on
Pagure, the message of schema topic ``pagure.issue.tag.added`` is created on the Fedora
Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_398-441>`__.
Removing label from an issue ticket
""""
+++++++++++++++++++++++++++++++++++
Whenever labels are removed from an existing issue ticket is created on a
project present on Pagure, the message of schema topic
``pagure.issue.tag.removed`` is created on the Fedora Messaging bus. The
class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_444-487>`_.
Whenever labels are removed from an existing issue ticket is created on a project
present on Pagure, the message of schema topic ``pagure.issue.tag.removed`` is created
on the Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/issue_schema.py#_444-487>`__.
Pull Requests
^^^^
~~~~~~~~~~~~~
The messages in this category follow the topic name pattern ``pagure.pull-request.*``.
Adding a pull request assignee
""""
++++++++++++++++++++++++++++++
Whenever an assignee is added to an existing pull request made against a
project on either Pagure or Pagure Dist Git, the message of schema topic
``pagure.pull-request.assigned.added`` is created on the Fedora Messaging bus.
The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_20-61>`_.
Whenever an assignee is added to an existing pull request made against a project on
either Pagure or Pagure Dist Git, the message of schema topic
``pagure.pull-request.assigned.added`` is created on the Fedora Messaging bus. The class
definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_20-61>`__.
Removing a pull request assignee
""""
++++++++++++++++++++++++++++++++
Whenever an assignee is removed from an existing pull request made against a
project on either Pagure or Pagure Dist Git, the message of schema topic
``pagure.pull-request.assigned.reset`` is created on the Fedora Messaging bus.
The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_64-104>`_.
Whenever an assignee is removed from an existing pull request made against a project on
either Pagure or Pagure Dist Git, the message of schema topic
``pagure.pull-request.assigned.reset`` is created on the Fedora Messaging bus. The class
definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_64-104>`__.
Closing a pull request
""""
++++++++++++++++++++++
Whenever an existing pull request made against a project on either Pagure or
Pagure Dist Git is closed, the message of schema topic
``pagure.pull-request.closed`` is created on the Fedora Messaging bus.
The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_107-151>`_.
Whenever an existing pull request made against a project on either Pagure or Pagure Dist
Git is closed, the message of schema topic ``pagure.pull-request.closed`` is created on
the Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_107-151>`__.
Adding a comment on a pull request
""""
++++++++++++++++++++++++++++++++++
Whenever a comment is added to an existing pull request made against a project
on either Pagure or Pagure Dist Git is closed, the message of schema topic
``pagure.pull-request.comment.added`` is created on the Fedora Messaging bus.
The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_154-200>`_.
Whenever a comment is added to an existing pull request made against a project on either
Pagure or Pagure Dist Git is closed, the message of schema topic
``pagure.pull-request.comment.added`` is created on the Fedora Messaging bus. The class
definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_154-200>`__.
Editing a comment on a pull request
""""
+++++++++++++++++++++++++++++++++++
Whenever a comment is edited on an existing pull request made against a project
on either Pagure or Pagure Dist Git is closed, the message of schema topic
``pagure.pull-request.comment.edited`` is created on the Fedora Messaging bus.
The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_203-249>`_.
Whenever a comment is edited on an existing pull request made against a project on
either Pagure or Pagure Dist Git is closed, the message of schema topic
``pagure.pull-request.comment.edited`` is created on the Fedora Messaging bus. The class
definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_203-249>`__.
Adding a flag on a pull request
""""
+++++++++++++++++++++++++++++++
Whenever a flag is added to an existing pull request made against a project
on either Pagure or Pagure Dist Git is closed, the message of schema topic
``pagure.pull-request.flag.added`` is created on the Fedora Messaging bus.
The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_252-293>`_.
Whenever a flag is added to an existing pull request made against a project on either
Pagure or Pagure Dist Git is closed, the message of schema topic
``pagure.pull-request.flag.added`` is created on the Fedora Messaging bus. The class
definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_252-293>`__.
Updating a flag on a pull request
""""
+++++++++++++++++++++++++++++++++
Whenever a flag is updated on an existing pull request made against a project
on either Pagure or Pagure Dist Git is closed, the message of schema topic
``pagure.pull-request.flag.updated`` is created on the Fedora Messaging bus.
The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_296-339>`_.
Whenever a flag is updated on an existing pull request made against a project on either
Pagure or Pagure Dist Git is closed, the message of schema topic
``pagure.pull-request.flag.updated`` is created on the Fedora Messaging bus. The class
definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_296-339>`__.
Editing a pull request body
""""
+++++++++++++++++++++++++++
Whenever the body of en existing pull request made against a project on either
Pagure or Pagure Dist Git is edited, the message of schema topic
``pagure.pull-request.initial_comment.edited`` is created on the Fedora
Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_342-384>`_.
Whenever the body of en existing pull request made against a project on either Pagure or
Pagure Dist Git is edited, the message of schema topic
``pagure.pull-request.initial_comment.edited`` is created on the Fedora Messaging bus.
The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_342-384>`__.
Creating a pull request
""""
+++++++++++++++++++++++
Whenever a pull request is made against a project on either Pagure or Pagure
Dist Git, the message of schema topic ``pagure.pull-request.new`` is created
on the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_387-427>`_.
Whenever a pull request is made against a project on either Pagure or Pagure Dist Git,
the message of schema topic ``pagure.pull-request.new`` is created on the Fedora
Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_387-427>`__.
Rebasing a pull request
""""
+++++++++++++++++++++++
Whenever a pull request made against a project on either Pagure or Pagure
Dist Git is rebased, the message of schema topic
``pagure.pull-request.rebased`` is created on the Fedora Messaging bus. The
class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_430-469>`_.
Whenever a pull request made against a project on either Pagure or Pagure Dist Git is
rebased, the message of schema topic ``pagure.pull-request.rebased`` is created on the
Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_430-469>`__.
Reopening a pull request
""""
++++++++++++++++++++++++
Whenever a closed pull request made against a project on either Pagure or
Pagure Dist Git is reopened, the message of schema topic
``pagure.pull-request.reopened`` is created on the Fedora Messaging bus. The
class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_472-511>`_.
Whenever a closed pull request made against a project on either Pagure or Pagure Dist
Git is reopened, the message of schema topic ``pagure.pull-request.reopened`` is created
on the Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_472-511>`__.
Adding a tag on a pull request
""""
++++++++++++++++++++++++++++++
Whenever a tag is added to an existing pull request made against a project on
either Pagure or Pagure Dist Git, the message of schema topic
``pagure.pull-request.tag.added`` is created on the Fedora Messaging bus. The
class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_514-556>`_.
Whenever a tag is added to an existing pull request made against a project on either
Pagure or Pagure Dist Git, the message of schema topic ``pagure.pull-request.tag.added``
is created on the Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_514-556>`__.
Removing a tag on a pull request
""""
++++++++++++++++++++++++++++++++
Whenever a tag is removed from an existing pull request made against a project
on either Pagure or Pagure Dist Git, the message of schema topic
``pagure.pull-request.tag.removed`` is created on the Fedora Messaging bus. The
class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_559-601>`_.
Whenever a tag is removed from an existing pull request made against a project on either
Pagure or Pagure Dist Git, the message of schema topic
``pagure.pull-request.tag.removed`` is created on the Fedora Messaging bus. The class
definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_559-601>`__.
Updating a pull request
""""
+++++++++++++++++++++++
Whenever an existing pull request made against a project on either Pagure or
Pagure Dist Git is updated, the message of schema topic
``pagure.pull-request.updated`` is created on the Fedora Messaging bus. The
class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_604-643>`_.
Whenever an existing pull request made against a project on either Pagure or Pagure Dist
Git is updated, the message of schema topic ``pagure.pull-request.updated`` is created
on the Fedora Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/pull_requests_schema.py#_604-643>`__.
Uncategorized
^^^^
~~~~~~~~~~~~~
The messages in this category do not follow any any certain topic name pattern.
Adding a commit flag
""""
++++++++++++++++++++
Whenever a commit flag is added in a project on either Pagure or Pagure Dist
Git, the message of schema topic ``pagure.commit.flag.added`` is created on the
Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/misc_schema.py#_20-69>`_.
Whenever a commit flag is added in a project on either Pagure or Pagure Dist Git, the
message of schema topic ``pagure.commit.flag.added`` is created on the Fedora Messaging
bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/misc_schema.py#_20-69>`__.
Updating a commit flag
""""
++++++++++++++++++++++
Whenever a commit flag is updated in a project on either Pagure or Pagure Dist
Git, the message of schema topic ``pagure.commit.flag.updated`` is created on
the Fedora Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/misc_schema.py#_72-121>`_.
Whenever a commit flag is updated in a project on either Pagure or Pagure Dist Git, the
message of schema topic ``pagure.commit.flag.updated`` is created on the Fedora
Messaging bus. The class definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/misc_schema.py#_72-121>`__.
Editing a group
""""
+++++++++++++++
Whenever the edits to a group are saved on either Pagure or Pagure Dist Git,
the message of schema topic ``pagure.group.edit`` is created on the Fedora
Messaging bus. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/misc_schema.py#_124-165>`_.
Whenever the edits to a group are saved on either Pagure or Pagure Dist Git, the message
of schema topic ``pagure.group.edit`` is created on the Fedora Messaging bus. The class
definition can be found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/misc_schema.py#_124-165>`__.
Sending a test notification
""""
+++++++++++++++++++++++++++
For debugging purposes, a test notification can be sent over with the schema
topic ``pagure.Test.notification`` is created on the Fedora Messaging bus. The
class can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/misc_schema.py#_168-198>`_.
For debugging purposes, a test notification can be sent over with the schema topic
``pagure.Test.notification`` is created on the Fedora Messaging bus. The class can be
found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/misc_schema.py#_168-198>`__.
Changes proposed
----
----------------
In order to switch out Pagure Dist Git for an alternative that provides a
relatively compatible API pattern, the following changes must be made in the
placed mentioned below.
In order to switch out Pagure Dist Git for an alternative that provides a relatively
compatible API pattern, the following changes must be made in the placed mentioned
below.
Necessary
^^^^
~~~~~~~~~
- The base class for all the topic classes ``PagureMessage`` has two member
variables storing information about the URL of the production (in
``__link__``) and staging (in ``__stg_link__``) deployments of either Pagure
or Pagure Dist Git. This needs to be changed in accordance with the newer
replacement planned. The class definition can be found
`here <https://pagure.io/pagure-messages/blob/master/f/pagure_messages/base.py#_401-443>`_.
- The base class for all the topic classes ``PagureMessage`` has two member variables
storing information about the URL of the production (in ``__link__``) and staging (in
``__stg_link__``) deployments of either Pagure or Pagure Dist Git. This needs to be
changed in accordance with the newer replacement planned. The class definition can be
found `here
<https://pagure.io/pagure-messages/blob/master/f/pagure_messages/base.py#_401-443>`__.
- The variable ``SCHEMA_URL`` currently leads to the page
``https://fedoraproject.org/message-schema/`` which does not exist. This
needs to be changed to point to the correct URL.
- There are multiple variables that contain the API schema for Pagure and
Pagure Dist Git, like ``TagColored``, ``BOARD``, ``BOARD_STATUS``,
``BOARD_ISSUE``, ``USER``, ``GIT_RECEIVE_USER``, ``PAGURE_LOG``,
``MILESTONES``, ``PRIORITIES``, ``BASE_PROJECT``, ``PROJECT``,
``RELATED_PR``, ``ISSUE``, ``PULL_REQUEST``, ``COMMIT_FLAG`` and ``GROUP``
that must undergo changes to store the API schemas for the replacement
platform.
``https://fedoraproject.org/message-schema/`` which does not exist. This needs to be
changed to point to the correct URL.
- There are multiple variables that contain the API schema for Pagure and Pagure Dist
Git, like ``TagColored``, ``BOARD``, ``BOARD_STATUS``, ``BOARD_ISSUE``, ``USER``,
``GIT_RECEIVE_USER``, ``PAGURE_LOG``, ``MILESTONES``, ``PRIORITIES``,
``BASE_PROJECT``, ``PROJECT``, ``RELATED_PR``, ``ISSUE``, ``PULL_REQUEST``,
``COMMIT_FLAG`` and ``GROUP`` that must undergo changes to store the API schemas for
the replacement platform.
- The schema project called ``pagure-messages`` need to be changed as well to
accommodate the JSON schema for the API of the alternative and hence, publish
messages accordingly.
accommodate the JSON schema for the API of the alternative and hence, publish messages
accordingly.
Unnecessary
^^^^
~~~~~~~~~~~
- The message classes themselves and the way they operate could, more or less,
stay the same as the way the messages are being sent to the Fedora Messaging
bus would remain the same.
- The message classes themselves and the way they operate could, more or less, stay the
same as the way the messages are being sent to the Fedora Messaging bus would remain
the same.

114
docs/dist-git-move/monitorgating.rst
View file

@ -1,98 +1,92 @@
.. _monitorgating:
##################################################
Pagure Dist Git Interactions With Monitor Gating
##################################################
Pagure Dist Git Interactions With Monitor Gating
================================================
`Monitor Gating <https://pagure.io/fedora-ci/monitor-gating>`_ is a set
of scripts that is used to test if `Fedora CI Gating
<https://docs.fedoraproject.org/en-US/ci/gating/>`_ is working as
expected. It essentially follows a typical workflow of cloning a package
from dist-git, rebuild it in koji, create a update in bodhi, then checks
that the CI runs, and the build is gated appropriately.
`Monitor Gating <https://pagure.io/fedora-ci/monitor-gating>`_ is a set of scripts that
is used to test if `Fedora CI Gating <https://docs.fedoraproject.org/en-US/ci/gating/>`_
is working as expected. It essentially follows a typical workflow of cloning a package
from dist-git, rebuild it in koji, create a update in bodhi, then checks that the CI
runs, and the build is gated appropriately.
***********************
Possible Interactions
***********************
Possible Interactions
---------------------
The following are a set of documented interactions possible between
Pagure Dist Git and Monitor Gating.
The following are a set of documented interactions possible between Pagure Dist Git and
Monitor Gating.
git and fedpkg CLI
==================
~~~~~~~~~~~~~~~~~~
**cloning a repo**
Monitor Gating uses ``fedpkg clone`` to clone a repo from dist-git.
Monitor Gating uses ``fedpkg clone`` to clone a repo from dist-git.
`utils.py#_107
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_107>`_
`utils.py#_107
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_107>`_
**adding a remote**
Monitor Gating uses ``git remote add`` to add a fork on dist-git as a
remote on a cloned dist-git repo.
Monitor Gating uses ``git remote add`` to add a fork on dist-git as a remote on a
cloned dist-git repo.
`utils.py#_136
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_136>`_
`utils.py#_136
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_136>`_
**switching branches**
Monitor Gating uses ``fedpkg switch-branch`` to switch branches on a
dist-git cloned repo.
Monitor Gating uses ``fedpkg switch-branch`` to switch branches on a dist-git cloned
repo.
`utils.py#_149
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_149>`_
`utils.py#_149
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_149>`_
**git pull, push, commit**
Monitor Gating uses ``git commit``, ``git pull```, and ``git push``
to get and make changes to a dist-git repo.
Monitor Gating uses ``git commit``, ``git pull```, and ``git push`` to get and make
changes to a dist-git repo.
`utils.py#_175-212
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_175-212>`_
`utils.py#_175-212
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_175-212>`_
dist-git (pagure) API
=====================
~~~~~~~~~~~~~~~~~~~~~
**CI status**
Monitor gating currently gets the status of the CI run(s) on a pull
request using the get PR flags endpoint
(``api/0/<namespace>/<name>/pull-request/<pr-id>/flag``).
Monitor gating currently gets the status of the CI run(s) on a pull request using
the get PR flags endpoint (``api/0/<namespace>/<name>/pull-request/<pr-id>/flag``).
`utils.py#_636
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_636>`_
`utils.py#_636
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_636>`_
**open pull request**
Monitor Gating opens new Pull Requests on dist-git using the pagure
API (``api/0/<namespace>/<name>/pull-request/new``)
Monitor Gating opens new Pull Requests on dist-git using the pagure API
(``api/0/<namespace>/<name>/pull-request/new``)
`utils.py#_214
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_214>`_
`utils.py#_214
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_214>`_
**merge pull request**
Monitor Gating merges Pull Requests on dist-git using the pagure API
(``api/0/<namespace>/<name>/pull-request/<pr-id>/merge``)
Monitor Gating merges Pull Requests on dist-git using the pagure API
(``api/0/<namespace>/<name>/pull-request/<pr-id>/merge``)
`utils.py#_691
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_691>`_
`utils.py#_691
<https://pagure.io/fedora-ci/monitor-gating/blob/production/f/monitor_gating/utils.py#_691>`_
******************
Changes proposed
******************
Changes proposed
----------------
In order to switch out Pagure Dist Git for an alternative that provides
a relatively compatible API pattern, the following changes must be made
in the places mentioned below.
In order to switch out Pagure Dist Git for an alternative that provides a relatively
compatible API pattern, the following changes must be made in the places mentioned
below.
Necessary
=========
~~~~~~~~~
- The changes made to the Continuous Integration component with regards
to the reporting of CI status need to be updated in monitor gating.
Currently dist-git uses the pagure flags system.
- Monitor Gating will need to be updated to support Opening and Merging
pull requests on the new solution
- The changes made to the Continuous Integration component with regards to the reporting
of CI status need to be updated in monitor gating. Currently dist-git uses the pagure
flags system.
- Monitor Gating will need to be updated to support Opening and Merging pull requests on
the new solution
Unnecessary
===========
~~~~~~~~~~~
- No changes should be needed for the basic git / fedpkg commands.
(this is based on the assumption the new solution is based on git)
- No changes should be needed for the basic git / fedpkg commands. (this is based on the
assumption the new solution is based on git)

151
docs/dist-git-move/notifications.rst
View file

@ -1,105 +1,98 @@
.. _notifications:
Pagure Dist Git Interactions With Fedora Notifications
====
======================================================
Fedora Messaging Notifier is a family of systems to manage end-user
notifications triggered by
`Fedora Messaging <https://fedora-messaging.readthedocs.io/>`_ and it provides
a single place for all applications using Fedora Messaging to notify users of
Fedora Messaging Notifier is a family of systems to manage end-user notifications
triggered by `Fedora Messaging <https://fedora-messaging.readthedocs.io/>`_ and it
provides a single place for all applications using Fedora Messaging to notify users of
events.
Possible Interactions
----
---------------------
The following are a set of documented interactions possible between Pagure Dist
Git and Fedora Notifications.
The following are a set of documented interactions possible between Pagure Dist Git and
Fedora Notifications.
Artifact Details
^^^^
~~~~~~~~~~~~~~~~
- Fedora Notifications receives information about the permissions related to
a certain asset (which can be a package) from Pagure Dist Git. It receives
information about the assets (which can be a package) that a user or a group
has access to.
- Fedora Notifications provide its users with **artifact details** via an HTTP
GET endpoint that it retrieves from Pagure Dist Git. For more information,
please check the
`function definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/api/handlers/misc.py#L55>`_
- Fedora Notifications receives information about the permissions related to a certain
asset (which can be a package) from Pagure Dist Git. It receives information about the
assets (which can be a package) that a user or a group has access to.
- Fedora Notifications provide its users with **artifact details** via an HTTP GET
endpoint that it retrieves from Pagure Dist Git. For more information, please check
the `function definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/api/handlers/misc.py#L55>`__
provided here.
- The inputs required by the Fedora Notifications service to perform this
operation are *name patterns of artifacts to be returned*,
*name of the users whose artifacts are to be returned*, and
*name of the groups whose artifacts are to be returned*.
- This endpoint makes use of a
`FastAPI dependency <https://fastapi.tiangolo.com/tutorial/dependencies/>`_
called ``get_distgit_proxy``, more information related to which can be found
`here <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L284>`_.
- The inputs required by the Fedora Notifications service to perform this operation are
*name patterns of artifacts to be returned*, *name of the users whose artifacts are to
be returned*, and *name of the groups whose artifacts are to be returned*.
- This endpoint makes use of a `FastAPI dependency
<https://fastapi.tiangolo.com/tutorial/dependencies/>`_ called ``get_distgit_proxy``,
more information related to which can be found `here
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L284>`__.
The ``PagureAsyncProxy`` class
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The URL of the existing Pagure Dist Git production deployment is referenced
here to instantiate an object of type ``PagureAsyncProxy`` class. For more
information, please check the
`method definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L69>`_
- The URL of the existing Pagure Dist Git production deployment is referenced here to
instantiate an object of type ``PagureAsyncProxy`` class. For more information, please
check the `method definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L69>`__
provided here.
- The
`method definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L72>`_
for **determining the next page parameters** is provided here. This is
extensively used for recursively navigating through the Pagure Dist Git
`API <https://pagure.io/api/0>`_ results.
- The
`method definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L86>`_
for **retrieving project details** is provided here. This is dependent on
the values provided for the ``namespace``, ``pattern``, ``username`` and
``owner`` parameters.
- The
`method definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L115>`_
for **retrieving project details associated with a certain user** is provided
here. This is dependent on the values provided for the ``username``
parameter.
- The
`method definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L157>`_
for **retrieving project details associated with a certain project** is
provided here. This is dependent on the values for the ``name`` parameter.
- The
`method definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L127>`_
for **retrieving user details associated with a certain project** is provided
here. This is dependent on the values provided for the ``project_path``
parameter.
- The
`method definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L142>`_
for **retrieving group details associated with a certain project** is provided
here. This is dependent on the values provided for the ``project_path``
parameter.
- The `method definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L72>`__ for
**determining the next page parameters** is provided here. This is extensively used
for recursively navigating through the Pagure Dist Git `API
<https://pagure.io/api/0>`_ results.
- The `method definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L86>`__ for
**retrieving project details** is provided here. This is dependent on the values
provided for the ``namespace``, ``pattern``, ``username`` and ``owner`` parameters.
- The `method definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L115>`__ for
**retrieving project details associated with a certain user** is provided here. This
is dependent on the values provided for the ``username`` parameter.
- The `method definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L157>`__ for
**retrieving project details associated with a certain project** is provided here.
This is dependent on the values for the ``name`` parameter.
- The `method definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L127>`__ for
**retrieving user details associated with a certain project** is provided here. This
is dependent on the values provided for the ``project_path`` parameter.
- The `method definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L142>`__ for
**retrieving group details associated with a certain project** is provided here. This
is dependent on the values provided for the ``project_path`` parameter.
Changes proposed
----
----------------
In order to switch out Pagure Dist Git for an alternative that provides a
relatively compatible API pattern, the following changes must be made in the
placed mentioned below.
In order to switch out Pagure Dist Git for an alternative that provides a relatively
compatible API pattern, the following changes must be made in the placed mentioned
below.
Necessary
^^^^
~~~~~~~~~
- The Pagure Dist Git URL production deployment URL would have to change to
point towards the production deployment URL of the alternative. This
`method definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L285>`_
is where the URL is formatted to point towards the API.
- The method ``get_api_url`` has to be redefined depending on what the URL
pattern is for the alternative place where we move Pagure Dist Git assets to.
Here is the
`method definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L69>`_
of where the changes need to be made.
- The Pagure Dist Git URL production deployment URL would have to change to point
towards the production deployment URL of the alternative. This `method definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L285>`__ is
where the URL is formatted to point towards the API.
- The method ``get_api_url`` has to be redefined depending on what the URL pattern is
for the alternative place where we move Pagure Dist Git assets to. Here is the `method
definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/pagure.py#L69>`__ of
where the changes need to be made.
Unnecessary
^^^^
~~~~~~~~~~~
- The ``APIClient`` class that is inherited by the ``PagureAsyncProxy`` class is
agnostic to the place where the assets are hosted so changing over from
Pagure Dist Git to an alternative would not necessitate a reworking while
checking for compatibilities is strongly recommended. The
`method definition <https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/base.py#L39>`_
can be found here.
agnostic to the place where the assets are hosted so changing over from Pagure Dist
Git to an alternative would not necessitate a reworking while checking for
compatibilities is strongly recommended. The `method definition
<https://github.com/fedora-infra/fmn/blob/develop/fmn/backends/base.py#L39>`__ can be
found here.

69
docs/dist-git-move/pagure.rst
View file

@ -1,62 +1,59 @@
.. _pagure:
Pagure Dist Git Interactions With Pagure
====
========================================
`Pagure <https://pagure.io/pagure>`_ is the git forge on which dist git is currently
based on. The dist git is a plugin for Pagure and it shares plenty of functionality with it.
`Pagure <https://pagure.io/pagure>`__ is the git forge on which dist git is currently
based on. The dist git is a plugin for Pagure and it shares plenty of functionality with
it.
Possible Interactions
----
---------------------
There aren't any interactions between Pagure and dist git in Fedora at all, but the
dist git is running on top of pagure.
As dist git is a plugin for Pagure, in this investigation only the dist git specific parts
will be mentioned.
There aren't any interactions between Pagure and dist git in Fedora at all, but the dist
git is running on top of pagure. As dist git is a plugin for Pagure, in this
investigation only the dist git specific parts will be mentioned.
Theme
^^^^
~~~~~
- The pagure contains the `theme <https://pagure.io/pagure/blob/master/f/pagure/themes/srcfpo>`_
for dist git
- The pagure contains the `theme
<https://pagure.io/pagure/blob/master/f/pagure/themes/srcfpo>`_ for dist git
Backend
^^^^
~~~~~~~
- Dist-git is using the `Pagure API <https://pagure.io/pagure/blob/master/f/pagure/api>`_
with few added endpoint, which could be recognized by having ``_dg/`` in URL
- Dist-git is using the same
`messaging scheme as pagure <https://github.com/Pagure/pagure-messages>`_
- Dist-git is using
`FAS integration <https://pagure.io/pagure/blob/master/f/pagure/ui/oidc_login.py>`_
from Pagure
- Dist-git is using
`ACL implementation <https://pagure.io/pagure/blob/master/f/pagure/lib/git_auth.py>`_
from pagure
- Dist-git is using the `Pagure API
<https://pagure.io/pagure/blob/master/f/pagure/api>`_ with few added endpoint, which
could be recognized by having ``_dg/`` in URL
- Dist-git is using the same `messaging scheme as pagure
<https://github.com/Pagure/pagure-messages>`_
- Dist-git is using `FAS integration
<https://pagure.io/pagure/blob/master/f/pagure/ui/oidc_login.py>`_ from Pagure
- Dist-git is using `ACL implementation
<https://pagure.io/pagure/blob/master/f/pagure/lib/git_auth.py>`_ from pagure
UI
^^^^
~~
- Dist-git is using
`Option to give repository <https://pagure.io/pagure/blob/master/f/pagure/ui/repo.py#_3171>`_
from pagure
- Dist-git is using `Option to give repository
<https://pagure.io/pagure/blob/master/f/pagure/ui/repo.py#_3171>`_ from pagure
- Issues are just link to bugzilla tracker for the repository in bugzilla dist-git
- Monitoring status setting (backend implemented in
`dist-git <https://pagure.io/pagure-dist-git/blob/master/f/pagure_distgit/plugin.py>`_)
- Orphaning and taking orphaned projects(backend implemented in
`dist-git <https://pagure.io/pagure-dist-git/blob/master/f/pagure_distgit/plugin.py>`_)
- Monitoring status setting (backend implemented in `dist-git
<https://pagure.io/pagure-dist-git/blob/master/f/pagure_distgit/plugin.py>`_)
- Orphaning and taking orphaned projects(backend implemented in `dist-git
<https://pagure.io/pagure-dist-git/blob/master/f/pagure_distgit/plugin.py>`_)
- Links to koji, bodhi, bugzilla, packages and koschei
Changes proposed
----
----------------
In order to switch out Pagure Dist Git for an alternative that provides a
relatively compatible API pattern, the following changes must be made in the
places mentioned below.
In order to switch out Pagure Dist Git for an alternative that provides a relatively
compatible API pattern, the following changes must be made in the places mentioned
below.
Necessary
^^^^
~~~~~~~~~
- The separate messaging schema needs to be created for dist git
- API needs to be separated from the pagure API

79
docs/dist-git-move/toddlers.rst
View file

@ -1,101 +1,104 @@
.. _toddlers:
Pagure Dist Git Interactions With Toddlers
====
==========================================
`Toddlers <https://pagure.io/fedora-infra/toddlers>`_ are used to run various small tasks
in Fedora Infrastructure. Keeping things synced and automating various tasks that were done
manually in the past. As that it has plenty of interactions with Dist Git.
`Toddlers <https://pagure.io/fedora-infra/toddlers>`__ are used to run various small
tasks in Fedora Infrastructure. Keeping things synced and automating various tasks that
were done manually in the past. As that it has plenty of interactions with Dist Git.
Possible Interactions
----
---------------------
The following are a set of documented interactions possible between Pagure Dist
Git and Toddlers.
The following are a set of documented interactions possible between Pagure Dist Git and
Toddlers.
Git module
^^^^
~~~~~~~~~~
- Some of the toddlers are doing cloning and various interactions with git, all of those
interactions are in
`git module <https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/utils/git.py>`_
interactions are in `git module
<https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/utils/git.py>`_
Pagure module
^^^^
~~~~~~~~~~~~~
- This `module <https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/utils/pagure.py>`_
- This `module
<https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/utils/pagure.py>`_
contains method interacting with pagure or dist-git
- Few methods are specific only for dist-git:
* ``set_monitoring_status``
* ``is_project_orphaned``
* ``assign_maintainer_to_project``
- ``set_monitoring_status``
- ``is_project_orphaned``
- ``assign_maintainer_to_project``
clean_retired_packages toddler
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- ``/_dg/orphan`` API endpoint called in ``orphan_package`` method
- ``/_dg/bzoverrides`` API endpoint called in ``orphan_package`` method
- ``/<namespace>/<name>/git/modifyacls`` API endpoint called in ``remove_access`` method
- ``/<namespace>/<name>/watchers/update`` API endpoint called in ``remove_access`` method
- ``/<namespace>/<name>/watchers/update`` API endpoint called in ``remove_access``
method
distgit_bugzilla_sync toddler
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- ``/extras/pagure_poc.json`` called in ``_get_pagure_projects``
- ``/extras/pagure_bz.json`` called in ``_get_pagure_projects``
flag_ci_pr toddler
^^^^
~~~~~~~~~~~~~~~~~~
- Listening to following topics:
* ``org.centos.*.ci.dist-git-pr.test.error``
* ``org.centos.*.ci.dist-git-pr.test.complete``
* ``org.centos.*.ci.dist-git-pr.test.running``
- ``org.centos.*.ci.dist-git-pr.test.error``
- ``org.centos.*.ci.dist-git-pr.test.complete``
- ``org.centos.*.ci.dist-git-pr.test.running``
- ``/api/0/<namespace>/<repo>/c/<commit_hash>/flag`` called in ``process`` method
flag_commit_build toddler
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~
- ``/api/0/<namespace>/<repo>/c/<commit_hash>/flag`` called in ``process`` method
packagers_without_bugzilla toddler
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- ``/extras/pagure_bz.json`` called in ``get_user_and_groups_dist_git`` method
pdc_retired_packages toddler
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- ``/<namespace>/<repo>/raw/<branch>/f/dead.package`` called in ``_is_retired_in_dist_git`` method
- ``/<namespace>/<repo>/raw/<branch>/f/dead.package`` called in
``_is_retired_in_dist_git`` method
pdc_unretire_packages toddler
^^^^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- ``/<namespace>/<package_name>.git`` called in ``process_ticket`` method
- ``git clone`` called in ``process_ticket`` toddler
Changes proposed
----
----------------
In order to switch out Pagure Dist Git for an alternative that provides a
relatively compatible API pattern, the following changes must be made in the
places mentioned below.
In order to switch out Pagure Dist Git for an alternative that provides a relatively
compatible API pattern, the following changes must be made in the places mentioned
below.
Necessary
^^^^
~~~~~~~~~
- The
`configuration <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/toddlers/templates/fedora-messaging.toml>`_
- The `configuration
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/toddlers/templates/fedora-messaging.toml>`_
needs to be changed for dist-git to point to new URL
- The new dist-git solution needs to provide the API calls needed by toddlers
Unnecessary
^^^^
~~~~~~~~~~~
- Split
`pagure module <https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/utils/pagure.py>`_
to dist-git and pagure module. This change will help us differentiate between those two
- Split `pagure module
<https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/utils/pagure.py>`_ to
dist-git and pagure module. This change will help us differentiate between those two
in the future

142
docs/dnf-countme/index.rst
View file

@ -4,123 +4,159 @@ DNF Countme
Purpose
-------
The purpose of this work was about investigating the current solution and it's bottlenecks to identify what needs to be done to solve the following problems:
The purpose of this work was about investigating the current solution and it's
bottlenecks to identify what needs to be done to solve the following problems:
* Storage bottleneck when creating the intermediate database file
* Operations efficiency for the infrastructure team
- Storage bottleneck when creating the intermediate database file
- Operations efficiency for the infrastructure team
Goals
-----
Short Term
**********
~~~~~~~~~~
The short term goal is about enhancing operational gaps and possible technical bottlenecks in the current solution.
The short term goal is about enhancing operational gaps and possible technical
bottlenecks in the current solution.
* Improve intermediate db file to consume less disk space
* Weekly data generation instead of daily
* Improve operations
- Improve intermediate db file to consume less disk space
- Weekly data generation instead of daily
- Improve operations
Long Term
*********
~~~~~~~~~
The long term goal aims to replace the current solution with an actual data driven application to provide end-user real time analytics (as close as possible) because of the following limitations:
The long term goal aims to replace the current solution with an actual data driven
application to provide end-user real time analytics (as close as possible) because of
the following limitations:
* Graphical reports are static images served through httpd
* Manual intervention is needed to generate reports outside of the cron job schedule
* Data is not real time
* There is no way to connect third party apps suchs as Jupyter since there is no “data service”
- Graphical reports are static images served through httpd
- Manual intervention is needed to generate reports outside of the cron job schedule
- Data is not real time
- There is no way to connect third party apps suchs as Jupyter since there is no “data
service”
The long term goal aims to create a data service and/or use an existing open source solution such as Prometheus/Kafka/etc. to serve that data from an API and a web app interface.
The long term goal aims to create a data service and/or use an existing open source
solution such as Prometheus/Kafka/etc. to serve that data from an API and a web app
interface.
The API would be useful for other apps to pull and filter “real time” data instead of downloading a sqlite db file to then parse it to useful human friendly formats.
The API would be useful for other apps to pull and filter “real time” data instead of
downloading a sqlite db file to then parse it to useful human friendly formats.
Resources
---------
* https://data-analysis.fedoraproject.org/csv-reports/countme/totals.db
* https://data-analysis.fedoraproject.org/csv-reports/mirrors/mirrorsdata-all.csv
* https://pagure.io/velociraptorizer
* https://pagure.io/brontosaurusifier/
- https://data-analysis.fedoraproject.org/csv-reports/countme/totals.db
- https://data-analysis.fedoraproject.org/csv-reports/mirrors/mirrorsdata-all.csv
- https://pagure.io/velociraptorizer
- https://pagure.io/brontosaurusifier/
Investigation
-------------
The investigation was about identifying possible bottenecks in the current solution, both technical and opertional.
The investigation was about identifying possible bottenecks in the current solution,
both technical and opertional.
The Current System
******************
~~~~~~~~~~~~~~~~~~
The current “system” is an ansible role[1] which relies on mirrors-countme and other scripts to do its job, most of those scripts are being executed from a cron job which generates static images that are served through a web server.
The current “system” is an ansible role[1] which relies on mirrors-countme and other
scripts to do its job, most of those scripts are being executed from a cron job which
generates static images that are served through a web server.
Someone from the Fedora infrastructure team needs to run that paybook if there is a need to run any of those tools outside of the cron job schedule which is quite limiting.
Someone from the Fedora infrastructure team needs to run that paybook if there is a need
to run any of those tools outside of the cron job schedule which is quite limiting.
[1] - https://pagure.io/fedora-infra/ansible/blob/main/f/roles/web-data-analysis
The Intermediate Database
*************************
~~~~~~~~~~~~~~~~~~~~~~~~~
The current process is that the script generates an intermediate database file, usually referred to as “raw.db”, so another file is created from this one (“totals.db”) which is used by end users.
The current process is that the script generates an intermediate database file, usually
referred to as “raw.db”, so another file is created from this one (“totals.db”) which is
used by end users.
The problem is that “raw.db” data is appended for each httpd Apache log line which is turning into a storage problem due to the increasing growth of that file size.
The problem is that “raw.db” data is appended for each httpd Apache log line which is
turning into a storage problem due to the increasing growth of that file size.
One possible solution that is on the table is to purge old data from “raw.db” every time the end user database file gets updated - for example: keep data from the last 30 days and delete everything else.
One possible solution that is on the table is to purge old data from “raw.db” every time
the end user database file gets updated - for example: keep data from the last 30 days
and delete everything else.
Another option is to create weekly database files for the intermediate/”raw.db” database, using the full year and week number as the filename, for example: YYYY/01.db instead of appending everything to one “raw.db” file - that would allow us to archive those files individually if needed.
Another option is to create weekly database files for the intermediate/”raw.db”
database, using the full year and week number as the filename, for example: YYYY/01.db
instead of appending everything to one “raw.db” file - that would allow us to archive
those files individually if needed.
Conclusions
-----------
We concluded that we have work with the current solution as a short term goal but should keep track of a system refactoring as a long term goal.
We concluded that we have work with the current solution as a short term goal but should
keep track of a system refactoring as a long term goal.
The `short term goal` is about removing storage bottlenecks and enhacing its operational effciency.
The `short term goal` is about removing storage bottlenecks and enhacing its operational
effciency.
The `long term goal` is about creating a data system that will replace the current solution entirely which may require another "arc initiative" as well.
The `long term goal` is about creating a data system that will replace the current
solution entirely which may require another "arc initiative" as well.
Proposed Roadmap
----------------
SOP
***
~~~
The team should write an SOP for the Fedora Infrastructure team about how and where data is generated.
The team should write an SOP for the Fedora Infrastructure team about how and where data
is generated.
The SOP Document should also describe the required steps to generate “on demand” data based on user request.
The SOP Document should also describe the required steps to generate “on demand” data
based on user request.
Data Generation
***************
~~~~~~~~~~~~~~~
The intermediate database file, also known as “raw.db”, is generated daily through a cron job.
The intermediate database file, also known as “raw.db”, is generated daily through a
cron job.
The cron job should be run weekly instead, because httpd logs are not “real time” and the system can suffer from eventual data losses by doing it daily.
The cron job should be run weekly instead, because httpd logs are not “real time” and
the system can suffer from eventual data losses by doing it daily.
This can be done by updating cron job file definitions in Fedora’s ansible repository: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/web-data-analysis/files
This can be done by updating cron job file definitions in Fedora’s ansible repository:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/web-data-analysis/files
Data Cleanup
************
~~~~~~~~~~~~
The intermediate “raw.db” file aggregates all parsed data from HTTPd logs which is turning into a storage problem on our log servers.
The intermediate “raw.db” file aggregates all parsed data from HTTPd logs which is
turning into a storage problem on our log servers.
There are two possible solutions for this problem: split database files based on “week of the year” or delete data from the intermediate database file that is older than 1 month.
There are two possible solutions for this problem: split database files based on “week
of the year” or delete data from the intermediate database file that is older than 1
month.
Splitting Database Files
^^^^^^^^^^^^^^^^^^^^^^^^
++++++++++++++++++++++++
This scheme would create a file per “week of the year” instead of a single intermediate database file.
This scheme would create a file per “week of the year” instead of a single intermediate
database file.
That would allow us to archive older files somewhere else while keeping the most recent ones in the server (the last 4 weeks for example).
That would allow us to archive older files somewhere else while keeping the most recent
ones in the server (the last 4 weeks for example).
This solution requires changes to how database files are written and the way we read those files to generate the final database file used by end users.
This solution requires changes to how database files are written and the way we read
those files to generate the final database file used by end users.
Database Cleanup
^^^^^^^^^^^^^^^^
++++++++++++++++
This approach would keep using a single “raw.db” database file but a new step would be added when adding data in the end user database file.
This approach would keep using a single “raw.db” database file but a new step would be
added when adding data in the end user database file.
The team would need to implement a step that would remove old data from the intermediate database file once the final counter database file is updated.
For example: read “raw.db” -> update “counter.db” -> delete all data from “raw.db” that is older than one month.
This approach is a bit simpler since it just needs an extra step in the existing code instead of changing how “raw.db” files are stored and used.
The team would need to implement a step that would remove old data from the intermediate
database file once the final counter database file is updated.
For example: read “raw.db” -> update “counter.db” -> delete all data from “raw.db” that
is older than one month.
This approach is a bit simpler since it just needs an extra step in the existing code
instead of changing how “raw.db” files are stored and used.

84
docs/fcas/creation_fail.rst
View file

@ -1,61 +1,53 @@
.. _creation_fail:
Why does the existing solution not work?
====
========================================
The existing solution does not work anymore and necessitates the need of a new
solution because of the following reasons.
The existing solution does not work anymore and necessitates the need of a new solution
because of the following reasons.
Internal reasons
----
----------------
The existing solution does not work anymore because of the following reasons
pertaining to the system itself.
The existing solution does not work anymore because of the following reasons pertaining
to the system itself.
1. Retrieval of a filtered list of human-only usernames from FASJSON service is
a performance intensive process and time consuming process both at the
serving and the receiving end. Having the service unit to fetch the list of
all usernames not only delivers penalty to the performance of the service
unit itself but also harms the experience of those using FASJSON service at
their ends. This can lead to severely unintended consequences such as
causing an unplanned downtime at the serving end, hogging up the reources
due to the contigious nature of the request and deteriorating the quality of
service for other FASJSON users.
2. Obtaining a moderated list of active human-only usernames from Datagrepper
is also a performance intensive and time consuming process both at the
serving and the receiving end. While running this service unit does not
deteriorate the quality of service of Datagrepper for other users, the
process is very slow as the Datagrepper service has to further interact with
the Datanommer database in order to be able to compute the request and fetch
the requested information elements. As a consequence, the service unit
runtime is very long and can be incredibly dependent on the network
1. Retrieval of a filtered list of human-only usernames from FASJSON service is a
performance intensive process and time consuming process both at the serving and the
receiving end. Having the service unit to fetch the list of all usernames not only
delivers penalty to the performance of the service unit itself but also harms the
experience of those using FASJSON service at their ends. This can lead to severely
unintended consequences such as causing an unplanned downtime at the serving end,
hogging up the reources due to the contigious nature of the request and deteriorating
the quality of service for other FASJSON users.
2. Obtaining a moderated list of active human-only usernames from Datagrepper is also a
performance intensive and time consuming process both at the serving and the
receiving end. While running this service unit does not deteriorate the quality of
service of Datagrepper for other users, the process is very slow as the Datagrepper
service has to further interact with the Datanommer database in order to be able to
compute the request and fetch the requested information elements. As a consequence,
the service unit runtime is very long and can be incredibly dependent on the network
performance of the client machine.
3. Having too many entities in the system not only makes it complicated but it
also brings in the added risks of failure and cost in time and effort
required to connect these entities with each other in the form of processes.
For instance, the service unit for fetching all the usernames from FASJSON
takes more than 90-120 minutes in one run and that for fetching all the
active usernames from Datagrepper takes more than 330-360 minutes in one
run. This severely restricts the opportunities for development of new
features and quality-of-life updates as the testing workflow could end up
easily taking a lot of time and resources.
3. Having too many entities in the system not only makes it complicated but it also
brings in the added risks of failure and cost in time and effort required to connect
these entities with each other in the form of processes. For instance, the service
unit for fetching all the usernames from FASJSON takes more than 90-120 minutes in
one run and that for fetching all the active usernames from Datagrepper takes more
than 330-360 minutes in one run. This severely restricts the opportunities for
development of new features and quality-of-life updates as the testing workflow could
end up easily taking a lot of time and resources.
External reasons
----
----------------
An new and improved solution is made necessary because of the following reasons
pertaining to the external entities.
1. In the wake of increased requirements from the stakeholders, the scope of
the tool widened drastically to a point that the existing solution could not
be extended to address the newer needs. While the design ideas and the
processes could be followed from the existing solution, the improved
solution was required to be developed to ensure it could handle larger data
elements, compute them to find out statistics about human-only users, amount
for multiple services that publish messages on the Fedora Messaging bus and
generate contribution statistics effectively. Therefore, an initiative was
proposed to better facilitate these needs.
1. In the wake of increased requirements from the stakeholders, the scope of the tool
widened drastically to a point that the existing solution could not be extended to
address the newer needs. While the design ideas and the processes could be followed
from the existing solution, the improved solution was required to be developed to
ensure it could handle larger data elements, compute them to find out statistics
about human-only users, amount for multiple services that publish messages on the
Fedora Messaging bus and generate contribution statistics effectively. Therefore, an
initiative was proposed to better facilitate these needs.

47
docs/fcas/creation_gram.rst
View file

@ -1,40 +1,31 @@
.. _creation_gram.rst:
Diagrams of Existing Solution
====
=============================
The existing solution for the problem statement came to address the previous
ticket which can be found
`here <https://pagure.io/fedora-infrastructure/issue/11149>`_. The project
repository is located at
`Fedora User Activity Statistics <https://github.com/t0xic0der/fuas>`_.
The existing solution for the problem statement came to address the previous ticket
which can be found `here <https://pagure.io/fedora-infrastructure/issue/11149>`_. The
project repository is located at `Fedora User Activity Statistics
<https://github.com/t0xic0der/fuas>`_.
The ``namelist`` unit
----
---------------------
Running on local device
^^^^
**Running on local device**
.. image:: ../_static/fcas-namelist-lo.png
:target: ../_images/fcas-namelist-lo.png
.. image:: ../_static/fcas-namelist-lo.png
:target: ../_images/fcas-namelist-lo.png
Running on GitHub Actions
^^^^
.. image:: ../_static/fcas-namelist-ga.png
:target: ../_images/fcas-namelist-ga.png
**Running on GitHub Actions**
.. image:: ../_static/fcas-namelist-ga.png
:target: ../_images/fcas-namelist-ga.png
The ``actvlist`` unit
----
---------------------
Running on local device
^^^^
**Running on local device**
.. image:: ../_static/fcas-actvlist-lo.png
:target: ../_images/fcas-actvlist-lo.png
.. image:: ../_static/fcas-actvlist-lo.png
:target: ../_images/fcas-actvlist-lo.png
Running on GitHub Actions
^^^^
.. image:: ../_static/fcas-actvlist-ga.png
:target: ../_images/fcas-actvlist-ga.png
**Running on GitHub Actions**
.. image:: ../_static/fcas-actvlist-ga.png
:target: ../_images/fcas-actvlist-ga.png

214
docs/fcas/creation_workflow.rst
View file

@ -1,28 +1,28 @@
.. _creation_workflow:
Existing solution
====
=================
The existing solution for the problem statement came to address the previous
ticket which can be found
`here <https://pagure.io/fedora-infrastructure/issue/11149>`_. The project
repository is located at
`Fedora User Activity Statistics <https://github.com/t0xic0der/fuas>`_.
The existing solution for the problem statement came to address the previous ticket
which can be found `here <https://pagure.io/fedora-infrastructure/issue/11149>`__. The
project repository is located at `Fedora User Activity Statistics
<https://github.com/t0xic0der/fuas>`_.
How does it work?
----
-----------------
The project consists of two main functional units: the ``namelist`` unit and
the ``actvlist`` unit. The ``namelist`` unit facilitates the retrieval of
usernames from the FASJSON service by the service runner, while the
``actvlist`` unit verifies the activity status of the names listed in the
aforementioned file through Datagrepper. Both units are executed as automated
cronjobs, scheduled to run at specific intervals. This ensures that the service
maintains an up-to-date list of usernames and a count of active users. The
service's behavior is controlled by a configurable file, allowing
administrators to tailor it according to their specific needs.
The project consists of two main functional units: the ``namelist`` unit and the
``actvlist`` unit. The ``namelist`` unit facilitates the retrieval of usernames from the
FASJSON service by the service runner, while the ``actvlist`` unit verifies the activity
status of the names listed in the aforementioned file through Datagrepper. Both units
are executed as automated cronjobs, scheduled to run at specific intervals. This ensures
that the service maintains an up-to-date list of usernames and a count of active users.
The service's behavior is controlled by a configurable file, allowing administrators to
tailor it according to their specific needs.
**Usage** ::
**Usage**
.. code-block::
Usage: fuas [OPTIONS] COMMAND [ARGS]...
@ -35,110 +35,112 @@ administrators to tailor it according to their specific needs.
namelist Fetch a list of usernames on the Fedora Account System
Configuration file
----
------------------
The sample configuration file can be found
`here <https://github.com/t0xic0der/fuas/blob/main/fuas/conf.py>`_ that can be
made a copy of and edited by the users to tailor-fit the service according to
their requirements.
The sample configuration file can be found `here
<https://github.com/t0xic0der/fuas/blob/main/fuas/conf.py>`__ that can be made a copy of
and edited by the users to tailor-fit the service according to their requirements.
The following is an exhaustive list of customizable variables. These variables
are intended to be customized by the users.
The following is an exhaustive list of customizable variables. These variables are
intended to be customized by the users.
1. ``daysqant`` (Default - 90) - Number of days for which the activity record is requested for
2. ``pagerows`` (Default - 1) - Number of rows to be displayed on a page when requesting data from Datagrepper
3. ``minactqt`` (Default - 5) - Minimum number of activities to be considered to count as user as "active"
4. ``services`` (Default - ["Pagure"]) - Services to probe into for activity records pertaining to the users
5. ``jsonloca`` (Default - "https://fasjson.fedoraproject.org") - Location where the FASJSON service is hosted
6. ``dgprlink`` (Default - "https://apps.fedoraproject.org/datagrepper/v2/search") - Location where the Datagrepper service is hosted
7. ``useriden`` (Default - "t0xic0der@FEDORAPROJECT.ORG") - User to masquerade as for probing into the FASJSON records
8. ``listlink`` (Default - "https://raw.githubusercontent.com/t0xic0der/fuas/main/data/namefile") - Location where the list of available users is present
9. ``namefile`` (Default - "/var/tmp/namefile") - Location where the list of available users is to be stored locally
10. ``actvfile`` (Default - "/var/tmp/actvfile") - Location where the list of active users is to be stored locally
11. ``acqtfile`` (Default - "/var/tmp/acqtfile") - Location where the count of active users is to be stored locally
1. ``daysqant`` (Default - 90) - Number of days for which the activity record is
requested for
2. ``pagerows`` (Default - 1) - Number of rows to be displayed on a page when requesting
data from Datagrepper
3. ``minactqt`` (Default - 5) - Minimum number of activities to be considered to count
as user as "active"
4. ``services`` (Default - ["Pagure"]) - Services to probe into for activity records
pertaining to the users
5. ``jsonloca`` (Default - "https://fasjson.fedoraproject.org") - Location where the
FASJSON service is hosted
6. ``dgprlink`` (Default - "https://apps.fedoraproject.org/datagrepper/v2/search") -
Location where the Datagrepper service is hosted
7. ``useriden`` (Default - "t0xic0der@FEDORAPROJECT.ORG") - User to masquerade as for
probing into the FASJSON records
8. ``listlink`` (Default -
"https://raw.githubusercontent.com/t0xic0der/fuas/main/data/namefile") - Location
where the list of available users is present
9. ``namefile`` (Default - "/var/tmp/namefile") - Location where the list of available
users is to be stored locally
10. ``actvfile`` (Default - "/var/tmp/actvfile") - Location where the list of active
users is to be stored locally
11. ``acqtfile`` (Default - "/var/tmp/acqtfile") - Location where the count of active
users is to be stored locally
The config file also consists of a list of computing variables of the global
scope. These variables are intended only for developers.
The config file also consists of a list of computing variables of the global scope.
These variables are intended only for developers.
1. ``dfltsize`` (Default - 1000) - Size of iterable pages for all entities present in the FASJSON service
1. ``dfltsize`` (Default - 1000) - Size of iterable pages for all entities present in
the FASJSON service
The ``namelist`` unit
----
---------------------
The service unit takes up the configuration variables like ``username`` and
``password`` for the user to masquerade as while probing into the FASJSON
service, ``jsonloca`` for getting the location where the FASJSON service is
hosted, ``namefile`` for storing the list of usernames received. Using a
session created with the masquerading user, the unit queries for the list of
all available users to FASJSON service and stores them into the file specified
in the configuration variable.
The service unit takes up the configuration variables like ``username`` and ``password``
for the user to masquerade as while probing into the FASJSON service, ``jsonloca`` for
getting the location where the FASJSON service is hosted, ``namefile`` for storing the
list of usernames received. Using a session created with the masquerading user, the unit
queries for the list of all available users to FASJSON service and stores them into the
file specified in the configuration variable.
The aforementioned session is created by using the ``krb5`` packages and the
``username`` and ``password`` are passed in the standard input of the console.
While this works for a smaller scale run where the said service unit is run in
ephemeral containers, this approach is highly discouraged and instead, a
session created using a keytab file is recommended in its stead. Also, a set of
workarounds must be placed in the default ``krb5`` configuration file to allow
for seamless authentication.
``username`` and ``password`` are passed in the standard input of the console. While
this works for a smaller scale run where the said service unit is run in ephemeral
containers, this approach is highly discouraged and instead, a session created using a
keytab file is recommended in its stead. Also, a set of workarounds must be placed in
the default ``krb5`` configuration file to allow for seamless authentication.
As this is a unit that runs for a longer period of time and makes queries that
are performance intensive in nature, it is strongly recommended to run this
unit no more than once or twice in a span of 24 hours. Also, it is essential to
ensure that the internet connection is reliable and the devices are not turned
off while the long-running service unit is in progress. This is owing to the
fact that the service unit is non-resumable in nature and writes to disk only
when the fetch is complete.
As this is a unit that runs for a longer period of time and makes queries that are
performance intensive in nature, it is strongly recommended to run this unit no more
than once or twice in a span of 24 hours. Also, it is essential to ensure that the
internet connection is reliable and the devices are not turned off while the
long-running service unit is in progress. This is owing to the fact that the service
unit is non-resumable in nature and writes to disk only when the fetch is complete.
To ensure a proper running of the service unit without any possible
interruptions, the service unit is run as a workflow on
`GitHub Actions <https://github.com/features/actions>`_. The workflow file can
be found here at
`https://github.com/t0xic0der/fuas/blob/main/.github/workflows/main.yml <https://github.com/t0xic0der/fuas/blob/main/.github/workflows/main.yml>`_
that helps to set up the environment for the service unit to run, fetches the
list of usernames and then commits them back to the same repository - making
that list publicly available for consumption. The time limit for running a
workflow on GitHub Actions is, however, 6 hours and that might, in some cases,
lead to timeouts and incomplete runs.
To ensure a proper running of the service unit without any possible interruptions, the
service unit is run as a workflow on `GitHub Actions
<https://github.com/features/actions>`_. The workflow file can be found here at
https://github.com/t0xic0der/fuas/blob/main/.github/workflows/main.yml that helps to set
up the environment for the service unit to run, fetches the list of usernames and then
commits them back to the same repository - making that list publicly available for
consumption. The time limit for running a workflow on GitHub Actions is, however, 6
hours and that might, in some cases, lead to timeouts and incomplete runs.
The ``actvlist`` unit
----
---------------------
The service unit takes up the configuration variables like ``listlink`` for
locating the file containing the list of all users registered on Fedora
Accounts System, ``daysqant`` for limiting the activity queries to under a said
number of days, ``minactqt`` for getting the bare minimum amount of activities
for a user to be counted as "active", ``services`` for looking into their
records for activities, ``dgprlink`` for getting the location where the
Datagrepper service is hosted, ``actvfile`` and ``acqtfile`` for storing the
names as well as counts of the active users respectively.
The service unit takes up the configuration variables like ``listlink`` for locating the
file containing the list of all users registered on Fedora Accounts System, ``daysqant``
for limiting the activity queries to under a said number of days, ``minactqt`` for
getting the bare minimum amount of activities for a user to be counted as "active",
``services`` for looking into their records for activities, ``dgprlink`` for getting the
location where the Datagrepper service is hosted, ``actvfile`` and ``acqtfile`` for
storing the names as well as counts of the active users respectively.
The service unit fetches the list of users from the aforementioned
configuration variables and iterates through them to find the activities
pertaining to the user in question. The period limit is appropriately set and
if the count of activities under the said period comes out to be greater than
or equal to the minimum number of activities decided - that user is considered
to be "active". Their username gets added to the list of all active users and
the count of active users is incremented accordingly. Both of these are stored
in the files specified in the configuration variables.
The service unit fetches the list of users from the aforementioned configuration
variables and iterates through them to find the activities pertaining to the user in
question. The period limit is appropriately set and if the count of activities under the
said period comes out to be greater than or equal to the minimum number of activities
decided - that user is considered to be "active". Their username gets added to the list
of all active users and the count of active users is incremented accordingly. Both of
these are stored in the files specified in the configuration variables.
As this is a unit that runs for a longer period of time and makes queries that
are performance intensive in nature, it is strongly recommended to run this
unit no more than once or twice in a span of 24 hours. Also, it is essential to
ensure that the internet connection is reliable and the devices are not turned
off while the long-running service unit is in progress. This is owing to the
fact that the service unit is non-resumable in nature and writes to disk only
when the fetch is complete. In an average, this service unit takes at least 4-6
times more time than the former service unit.
To ensure a proper running of the service unit without any possible hiccups,
the service unit is run as a workflow on
`GitHub Actions <https://github.com/features/actions>`_. The workflow file can
be found here at
`https://github.com/t0xic0der/fuas/blob/main/.github/workflows/actv.yml <https://github.com/t0xic0der/fuas/blob/main/.github/workflows/actv.yml>`_
than helps to set up the environment for the service unit to run, fetches the
list of active usernames as well as the count and then commits them back to the
repository - making that list as well as the count publicly available for
consumption. The time limit for running a workflow on GitHub Actions is,
however, 6 hours and that might, in some cases, lead to timeouts and incomplete
runs.
As this is a unit that runs for a longer period of time and makes queries that are
performance intensive in nature, it is strongly recommended to run this unit no more
than once or twice in a span of 24 hours. Also, it is essential to ensure that the
internet connection is reliable and the devices are not turned off while the
long-running service unit is in progress. This is owing to the fact that the service
unit is non-resumable in nature and writes to disk only when the fetch is complete. In
an average, this service unit takes at least 4-6 times more time than the former service
unit.
To ensure a proper running of the service unit without any possible hiccups, the service
unit is run as a workflow on `GitHub Actions <https://github.com/features/actions>`_.
The workflow file can be found here at
https://github.com/t0xic0der/fuas/blob/main/.github/workflows/actv.yml than helps to set
up the environment for the service unit to run, fetches the list of active usernames as
well as the count and then commits them back to the repository - making that list as
well as the count publicly available for consumption. The time limit for running a
workflow on GitHub Actions is, however, 6 hours and that might, in some cases, lead to
timeouts and incomplete runs.

202
docs/fcas/index.rst
View file

@ -1,96 +1,94 @@
Fedora Contributor Activity Statistics
====
======================================
Purpose
----
-------
In order to have a quantitative understanding of how the contributor activity
has changed over the years and to provide the foundational support to the
Fedora Project strategy 2028's guiding star about doubling the number of active
contributors every week, it is important to have a service that tracks their
statistics. This measurement would help make the strategy goal meaningful as
well as assist the Fedora Council and the related bodies understand how far
they have progressed into making this happen and identify the underlying
particular problems that act as a barrier in realizing this objective.
In order to have a quantitative understanding of how the contributor activity has
changed over the years and to provide the foundational support to the Fedora Project
strategy 2028's guiding star about doubling the number of active contributors every
week, it is important to have a service that tracks their statistics. This measurement
would help make the strategy goal meaningful as well as assist the Fedora Council and
the related bodies understand how far they have progressed into making this happen and
identify the underlying particular problems that act as a barrier in realizing this
objective.
Background
----
----------
There was a
`Fedora Council <https://docs.fedoraproject.org/en-US/council/>`_
Face To Face 2023 Hackfest organized in Frankfurt,
Germany that was attended by the Fedora Council members,
`Akashdeep Dhar <https://accounts.fedoraproject.org/user/t0xic0der>`_,
`Alexandra Fedorova <https://accounts.fedoraproject.org/user/bookwar>`_,
`Ben Cottom <https://accounts.fedoraproject.org/user/bcotton>`_,
`David Cantrell <https://accounts.fedoraproject.org/user/dcantrell>`_,
`Justin W. Flory <https://accounts.fedoraproject.org/user/jflory7>`_,
`Matthew Miller <https://accounts.fedoraproject.org/user/mattdm>`_,
`Sumantro Mukherjee <https://accounts.fedoraproject.org/user/sumantrom>`_ and
`Vipul Siddharth <https://accounts.fedoraproject.org/user/siddharthvipul1>`_.
Among a bunch of strategy goals discussed about and decided upon there, the
core driving goal for the five-year strategy plan was to facilitate a
community environment where the number of active contributors double up every
week.
There was a `Fedora Council <https://docs.fedoraproject.org/en-US/council/>`_ Face To
Face 2023 Hackfest organized in Frankfurt, Germany that was attended by the Fedora
Council members, `Akashdeep Dhar <https://accounts.fedoraproject.org/user/t0xic0der>`_,
`Alexandra Fedorova <https://accounts.fedoraproject.org/user/bookwar>`_, `Ben Cottom
<https://accounts.fedoraproject.org/user/bcotton>`_, `David Cantrell
<https://accounts.fedoraproject.org/user/dcantrell>`_, `Justin W. Flory
<https://accounts.fedoraproject.org/user/jflory7>`_, `Matthew Miller
<https://accounts.fedoraproject.org/user/mattdm>`_, `Sumantro Mukherjee
<https://accounts.fedoraproject.org/user/sumantrom>`_ and `Vipul Siddharth
<https://accounts.fedoraproject.org/user/siddharthvipul1>`_. Among a bunch of strategy
goals discussed about and decided upon there, the core driving goal for the five-year
strategy plan was to facilitate a community environment where the number of active
contributors double up every week.
This was previously proposed as an Fedora Infrastructure
`ticket <https://pagure.io/fedora-infrastructure/issue/11149>`_ by
`Michal Konecny <https://accounts.fedoraproject.org/user/zlopez>`_ on Matthew
Miller's request and addressed by Akashdeep Dhar in the project called
`Fedora User Activity Statistics <https://github.com/t0xic0der/fuas>`_.
During the `Community Platform Engineering <https://docs.fedoraproject.org/en-US/cpe/>`_
`Face To Face Meeting 2023 <https://fedoramagazine.org/the-community-platform-engineering-f2f-2023-experience-part-i/>`_
in Barcelona, Spain - the scope of the project was revisited by Akashdeep Dhar,
`Adam Saleh <https://accounts.fedoraproject.org/user/asaleh>`_,
`David Kirwan <https://accounts.fedoraproject.org/user/dkirwan>`_,
`Kevin Fenzi <https://accounts.fedoraproject.org/user/nirik>`_ and
Matthew Miller which led to the refinement of the projects purpose and an
increase in the deliverable requirements.
This was previously proposed as an Fedora Infrastructure `ticket
<https://pagure.io/fedora-infrastructure/issue/11149>`_ by `Michal Konecny
<https://accounts.fedoraproject.org/user/zlopez>`_ on Matthew Miller's request and
addressed by Akashdeep Dhar in the project called `Fedora User Activity Statistics
<https://github.com/t0xic0der/fuas>`_. During the `Community Platform Engineering
<https://docs.fedoraproject.org/en-US/cpe/>`_ `Face To Face Meeting 2023
<https://fedoramagazine.org/the-community-platform-engineering-f2f-2023-experience-part-i/>`_
in Barcelona, Spain - the scope of the project was revisited by Akashdeep Dhar, `Adam
Saleh <https://accounts.fedoraproject.org/user/asaleh>`_, `David Kirwan
<https://accounts.fedoraproject.org/user/dkirwan>`_, `Kevin Fenzi
<https://accounts.fedoraproject.org/user/nirik>`_ and Matthew Miller which led to the
refinement of the projects purpose and an increase in the deliverable requirements.
Following the expanded scope of the project, the previously provided solution
no longer addressed the updated set of requirements. Adam Saleh and Akashdeep
Dhar had a discussion about efficient methods of extracting information from
the Datanommer service. The project was proposed to be an initiative in
`this ticket <https://pagure.io/cpe/initiatives-proposal/issue/27>`_ by
`Aoife Moloney <https://accounts.fedoraproject.org/user/amoloney>`_. The
project was then scoped for ARC investigation for the period of Q2 2023 before
it is sent for implementation by the respective initiative team assigned to the
said project.
Following the expanded scope of the project, the previously provided solution no longer
addressed the updated set of requirements. Adam Saleh and Akashdeep Dhar had a
discussion about efficient methods of extracting information from the Datanommer
service. The project was proposed to be an initiative in `this ticket
<https://pagure.io/cpe/initiatives-proposal/issue/27>`_ by `Aoife Moloney
<https://accounts.fedoraproject.org/user/amoloney>`_. The project was then scoped for
ARC investigation for the period of Q2 2023 before it is sent for implementation by the
respective initiative team assigned to the said project.
Functional requirements
----
-----------------------
The following section details about the requirements for the project in both
aspects - the bare minimum outcome to be able to call the project as success
as well as the list of nice-to-have wishes that constitute the absolute maximum
outcome. Please note that these requirements must be taken as recommendations
and changes introduced to them during the implementation phase of the project
when bound by the circumstances is acceptable.
The following section details about the requirements for the project in both aspects -
the bare minimum outcome to be able to call the project as success as well as the list
of nice-to-have wishes that constitute the absolute maximum outcome. Please note that
these requirements must be taken as recommendations and changes introduced to them
during the implementation phase of the project when bound by the circumstances is
acceptable.
Minimal
^^^^
~~~~~~~
* Processing - A collector service for legitimate human-owned/run accounts
* Output - Statistical information created in JSON format
- Processing - A collector service for legitimate human-owned/run accounts
- Output - Statistical information created in JSON format
Maximal
^^^^
~~~~~~~
* Processing - Analyzing activity from meetbot logs
* Output - Report automatically being generated on a weekly basis
- Processing - Analyzing activity from meetbot logs
- Output - Report automatically being generated on a weekly basis
Resources
----
---------
* `Fedora User Activity Statistics <https://github.com/t0xic0der/fuas>`_
* `Datagrepper <https://apps.fedoraproject.org/datagrepper/>`_
* `Monitor Dashboard <https://monitor-dashboard-monitor-gating.apps.ocp.fedoraproject.org/>`_
* `Datanommer <https://github.com/fedora-infra/datanommer>`_
* `Original Fedora Infrastructure ticket <https://pagure.io/fedora-infrastructure/issue/11149>`_
* `Renewed Initiative Proposal ticket <https://pagure.io/cpe/initiatives-proposal/issue/27>`_
- `Fedora User Activity Statistics <https://github.com/t0xic0der/fuas>`_
- `Datagrepper <https://apps.fedoraproject.org/datagrepper/>`_
- `Monitor Dashboard
<https://monitor-dashboard-monitor-gating.apps.ocp.fedoraproject.org/>`_
- `Datanommer <https://github.com/fedora-infra/datanommer>`_
- `Original Fedora Infrastructure ticket
<https://pagure.io/fedora-infrastructure/issue/11149>`_
- `Renewed Initiative Proposal ticket
<https://pagure.io/cpe/initiatives-proposal/issue/27>`_
Index
----
-----
.. toctree::
:maxdepth: 2
@ -105,43 +103,41 @@ Index
solution_techtool
Conclusions
----
-----------
After understanding how effective the project can be in helping the Fedora
Council achieve its strategic objective of doubling the number of active
contributors present over a given period of time, the options for making the
said service as useful as possible were explored. It was concluded that the
historical data collected by the Datanommer from the Fedora Messaging bus
would be indeed helpful in tracking contribution activities and detailing on
contribution statistics and that it should be theoretically possible for the
team to implement such a service.
After understanding how effective the project can be in helping the Fedora Council
achieve its strategic objective of doubling the number of active contributors present
over a given period of time, the options for making the said service as useful as
possible were explored. It was concluded that the historical data collected by the
Datanommer from the Fedora Messaging bus would be indeed helpful in tracking
contribution activities and detailing on contribution statistics and that it should be
theoretically possible for the team to implement such a service.
Roadmap
----
-------
- **Step 1** - Connect with the data scientists to understand which data
elements need to be focused on
- **Step 2** - Author codebase to obtain details on human-run and human-owned
legitimate accounts
- **Step 3** - Author SQL queries for obtaining historical contribution
statistics per username
- **Step 4** - Author SQL queries for obtaining historical contribution
statistics per service
- **Step 5** - Adapt the queries to create a service to obtain current and
future statistics
- **Step 6** - Expose necessary endpoints or integrations on the dashboard for
the analytics
- **Step 7** - Setup the staging environment for the dashboard in a limited
testing environment for inspection
- **Step 8** - Deploy to the production environment after ironing out the
vertex cases for statistics and... PROFIT?
- **Step 1** - Connect with the data scientists to understand which data elements need
to be focused on
- **Step 2** - Author codebase to obtain details on human-run and human-owned legitimate
accounts
- **Step 3** - Author SQL queries for obtaining historical contribution statistics per
username
- **Step 4** - Author SQL queries for obtaining historical contribution statistics per
service
- **Step 5** - Adapt the queries to create a service to obtain current and future
statistics
- **Step 6** - Expose necessary endpoints or integrations on the dashboard for the
analytics
- **Step 7** - Setup the staging environment for the dashboard in a limited testing
environment for inspection
- **Step 8** - Deploy to the production environment after ironing out the vertex cases
for statistics and... PROFIT?
Estimate of work
----
----------------
As this service makes active use of technologies that are already created and
maintained such as Fedora Messaging, Datagrepper, Datanommer, FASJSON etc.,
and assuming that the team that is to work on this down the road has people who
are experienced in the aforementioned technologies, the service should not take
any longer than two quarters to hit the staging environment and one more
quarter to make it to the production one.
As this service makes active use of technologies that are already created and maintained
such as Fedora Messaging, Datagrepper, Datanommer, FASJSON etc., and assuming that the
team that is to work on this down the road has people who are experienced in the
aforementioned technologies, the service should not take any longer than two quarters to
hit the staging environment and one more quarter to make it to the production one.

156
docs/fcas/solution_dataeplt.rst
View file

@ -1,14 +1,14 @@
.. _solution_dataeplt.rst:
Data Exploration and Significance
====
=================================
The following is a set of information that would be looked into by the said
service whenever it would be deployed. Please note that this list consists
of both - the information that would be available for consumption by the
service users as well as the information that would be available for
computation and analysis to the service itself but not the service users, and
there can be more such information apart from the ones listed below.
The following is a set of information that would be looked into by the said service
whenever it would be deployed. Please note that this list consists of both - the
information that would be available for consumption by the service users as well as the
information that would be available for computation and analysis to the service itself
but not the service users, and there can be more such information apart from the ones
listed below.
1. Activity entry from Datanommer (For computation only)
2. Username of the "subject" i.e. owner of the contribution (For computation only)
@ -19,102 +19,94 @@ there can be more such information apart from the ones listed below.
7. Service where a grouped contribution activity happened (For consumption only)
8. Activity trends per username (For computation only)
Activity Entry from Datanommer
----
------------------------------
This data forms the most basic functional entity of a "contribution record". An
occurrence of an activity means that a contribution was made by the "subject"
member on the "object" member and/or service with the "predicament" nature of
the contribution at the "time" of it happening. A computed collection of these
data can help form wider statistics for example - trend of contribution by a
certain "subject" member, trend of contribution on a certain "service" etc.
allowing us to answer questions like "which services are most active (and why)
and least active (any why)?", "what period of time attracts most contributions
(and why)?" etc. As this data is intricate, it only serves its purpose when a
computed group of those form statistics and not when it is singled out - and
that is why this data is only used for computational purposes only.
occurrence of an activity means that a contribution was made by the "subject" member on
the "object" member and/or service with the "predicament" nature of the contribution at
the "time" of it happening. A computed collection of these data can help form wider
statistics for example - trend of contribution by a certain "subject" member, trend of
contribution on a certain "service" etc. allowing us to answer questions like "which
services are most active (and why) and least active (any why)?", "what period of time
attracts most contributions (and why)?" etc. As this data is intricate, it only serves
its purpose when a computed group of those form statistics and not when it is singled
out - and that is why this data is only used for computational purposes only.
Username of the "subject"
----
-------------------------
Alternatively, owner of the contribution.
This data is a part of the previously-stated "activity entry from Datanommer"
data. In order to protect the privacy of the members involved in the
aforementioned data, this information is anonymized as a hash and due to the
fact that this data serves its purpose when a computed group of those form
statistics and not when it is singled out - this data is only used for
computational purposes only.
This data is a part of the previously-stated "activity entry from Datanommer" data. In
order to protect the privacy of the members involved in the aforementioned data, this
information is anonymized as a hash and due to the fact that this data serves its
purpose when a computed group of those form statistics and not when it is singled out -
this data is only used for computational purposes only.
Username of the "object"
----
------------------------
Alternatively, involved in the contribution.
This data is a part of the previously-stated "activity entry from Datanommer"
data. In order to protect the privacy of the members involved in the
aforementioned data, this information is anonymized as a hash and due to the
fact that this data serves its purpose when a computed group of those form
statistics and not when it is singled out - this data is only used for
computational purposes only.
This data is a part of the previously-stated "activity entry from Datanommer" data. In
order to protect the privacy of the members involved in the aforementioned data, this
information is anonymized as a hash and due to the fact that this data serves its
purpose when a computed group of those form statistics and not when it is singled out -
this data is only used for computational purposes only.
Datetime data of a specific contribution activity
----
-------------------------------------------------
This data is a part of the previously-stated "activity entry from Datanommer"
data. Due to the fact that this data serves its purpose when a computed group
of those form statistics and not when it is singled out - this data is only
used for computational purposes only.
This data is a part of the previously-stated "activity entry from Datanommer" data. Due
to the fact that this data serves its purpose when a computed group of those form
statistics and not when it is singled out - this data is only used for computational
purposes only.
Datetime data of a grouped contribution actvitity
----
-------------------------------------------------
Being a derivative statistic obtained from a computed group of the previously
stated "datetime of a specific contribution activity", this can be used to
understand the trend of contribution over a period of "time" for contributions
of a certain "nature", contributions over a period of "time" for contributions
on a certain "service" etc. This understanding would help us answer questions
like what timelines attract most contributions, what timelines do not have much
of contributions etc. and gauge the success of activities such as events and
workshops by helping answer if those were able to bring in contributions right
after their commencement time. As a result, this data is available for user
consumption by the service.
Being a derivative statistic obtained from a computed group of the previously stated
"datetime of a specific contribution activity", this can be used to understand the trend
of contribution over a period of "time" for contributions of a certain "nature",
contributions over a period of "time" for contributions on a certain "service" etc. This
understanding would help us answer questions like what timelines attract most
contributions, what timelines do not have much of contributions etc. and gauge the
success of activities such as events and workshops by helping answer if those were able
to bring in contributions right after their commencement time. As a result, this data is
available for user consumption by the service.
Service where a specific contribution activity happened
----
-------------------------------------------------------
This data is a part of the previously-stated "activity entry from Datanommer"
data. As this data is intricate, it only serves its purpose when a computed
group of those form statistics and not when it is singled out - and that is why
this data is only used for computational purposes only.
Service where a grouped contribution activity happened
----
Being a derivative statistic obtained from a computed group of the previously
stated "service where a specific grouped contribution activity happened", this
can be used to understand the trend of contribution on a certain service and
create comparisons of those against another to see how they fare in the
contribution activities. This understanding would help us answer questions like
what services are most active in terms of contributons and what services are
not and gauge the usability of those services by knowing what makes those
services desirable (i.e. inferred from favourable contribution statistics) and
undesirable (i.e. inferred from unfavourable contribution statistics) to direct
what service to be contributed to. As a result, this data is available for user
consumptions by the service.
Activity trends per username
----
Being a derivative statistic obtained from a computed group of the previously
stated "activity entry from Datanommer", this can be used to understand the
trend of contribution for a certain user. This understanding would help us
answer questions like what fields a certain member contributes to and if they
are transitioning from one field to another, what reasons have led them to do
that. In order to protect the privacy of the members involved in the
aforementioned data, this information is anonymized as a hash and due to the
fact that this data serves its purpose when a computed group of those form
statistics and not when it is singled out - this data is only used for
This data is a part of the previously-stated "activity entry from Datanommer" data. As
this data is intricate, it only serves its purpose when a computed group of those form
statistics and not when it is singled out - and that is why this data is only used for
computational purposes only.
Service where a grouped contribution activity happened
------------------------------------------------------
Being a derivative statistic obtained from a computed group of the previously stated
"service where a specific grouped contribution activity happened", this can be used to
understand the trend of contribution on a certain service and create comparisons of
those against another to see how they fare in the contribution activities. This
understanding would help us answer questions like what services are most active in terms
of contributons and what services are not and gauge the usability of those services by
knowing what makes those services desirable (i.e. inferred from favourable contribution
statistics) and undesirable (i.e. inferred from unfavourable contribution statistics) to
direct what service to be contributed to. As a result, this data is available for user
consumptions by the service.
Activity trends per username
----------------------------
Being a derivative statistic obtained from a computed group of the previously stated
"activity entry from Datanommer", this can be used to understand the trend of
contribution for a certain user. This understanding would help us answer questions like
what fields a certain member contributes to and if they are transitioning from one field
to another, what reasons have led them to do that. In order to protect the privacy of
the members involved in the aforementioned data, this information is anonymized as a
hash and due to the fact that this data serves its purpose when a computed group of
those form statistics and not when it is singled out - this data is only used for
computational purposes only.

82
docs/fcas/solution_datanote.rst
View file

@ -1,62 +1,54 @@
.. _solution_datanote.rst:
Planned Statistics Sources
====
==========================
For obtaining the list of all usernames
----
---------------------------------------
In contrast to the earlier approach of gathering statistics for the active
users available on FASJSON, this time around the statistics would need to be a
lot more verbose and detailed. The core part that binds these statistics is the
nature of the information collected as we would want to ensure that these
activities are "objectively" perceived. What looks like a contribution to a
certain person should look the same to others as well.
In contrast to the earlier approach of gathering statistics for the active users
available on FASJSON, this time around the statistics would need to be a lot more
verbose and detailed. The core part that binds these statistics is the nature of the
information collected as we would want to ensure that these activities are "objectively"
perceived. What looks like a contribution to a certain person should look the same to
others as well.
The list of the all available users has been planned to be resourced from the
authentication database itself, instead of interacting with an HTTP API like
FASJSON first and then letting FASJSON connect with the authentication database
like we did before. This is to ensure that the scheduled runs for the process to
fetch and update the list of users is not performance intensive and lesser
time consuming in general than it was before.
authentication database itself, instead of interacting with an HTTP API like FASJSON
first and then letting FASJSON connect with the authentication database like we did
before. This is to ensure that the scheduled runs for the process to fetch and update
the list of users is not performance intensive and lesser time consuming in general than
it was before.
Previous architecture
^^^^
**Previous architecture**
.. image:: ../_static/fcas-namelist-prev.png
:target: ../_images/fcas-namelist-prev.png
.. image:: ../_static/fcas-namelist-prev.png
:target: ../_images/fcas-namelist-prev.png
Planned architecture
^^^^
.. image:: ../_static/fcas-namelist-curt.png
:target: ../_images/fcas-namelist-curt.png
**Planned architecture**
.. image:: ../_static/fcas-namelist-curt.png
:target: ../_images/fcas-namelist-curt.png
For obtaining the list of active usernames
----
------------------------------------------
`Fedora Messaging <https://fedora-messaging.readthedocs.io/>`_ keeps track of
the activities performed on a certain set of services deployed on the community
`Fedora Messaging <https://fedora-messaging.readthedocs.io/>`_ keeps track of the
activities performed on a certain set of services deployed on the community
infrastructure. As the act of making contributions would inevitably require the
interactions with these services - it is very likely that the contribution
activities would cause the service to publish a message on the Fedora Messaging
bus, should the relevant schemas be configured correctly.
interactions with these services - it is very likely that the contribution activities
would cause the service to publish a message on the Fedora Messaging bus, should the
relevant schemas be configured correctly.
The historical collection of all messages published on the Fedora Messaging bus
since the time of its deployment in production has been stored on a database
service known as `Datanommer <https://github.com/fedora-infra/datanommer>`_ and
this is something that can be probed into directly without including another
element in the system for facilitating the requests, which was previously done
by `Datagrepper <https://apps.fedoraproject.org/datagrepper/>`_.
The historical collection of all messages published on the Fedora Messaging bus since
the time of its deployment in production has been stored on a database service known as
`Datanommer <https://github.com/fedora-infra/datanommer>`_ and this is something that
can be probed into directly without including another element in the system for
facilitating the requests, which was previously done by `Datagrepper
<https://apps.fedoraproject.org/datagrepper/>`_.
Previous architecture
^^^^
**Previous architecture**
.. image:: ../_static/fcas-actvlist-prev.png
:target: ../_images/fcas-actvlist-prev.png
.. image:: ../_static/fcas-actvlist-prev.png
:target: ../_images/fcas-actvlist-prev.png
Planned architecture
^^^^
.. image:: ../_static/fcas-actvlist-curt.png
:target: ../_images/fcas-actvlist-curt.png
**Planned architecture**
.. image:: ../_static/fcas-actvlist-curt.png
:target: ../_images/fcas-actvlist-curt.png

266
docs/fcas/solution_examples.rst
View file

@ -1,227 +1,181 @@
.. _solution_examples.rst:
Examples of Contributor Actitivies
====
==================================
Below is an extensive compilation of sample activities that qualify as
"contributions" and can be systematically monitored. These activities involve
engaging with services responsible for publishing messages on the Fedora
Messaging bus. For better organization, these examples can be categorized
according to the specific services they are associated with. It is important to
recognize that this list is not exhaustive, and there may be additional
examples beyond those provided here.
Below is an extensive compilation of sample activities that qualify as "contributions"
and can be systematically monitored. These activities involve engaging with services
responsible for publishing messages on the Fedora Messaging bus. For better
organization, these examples can be categorized according to the specific services they
are associated with. It is important to recognize that this list is not exhaustive, and
there may be additional examples beyond those provided here.
COPR
----
Application identifier - ``Copr``
Tasks
^^^^
**Tasks**
- Creating/modifying/deleting package namespaces or primitives
- Pushing/updating/removing builds/tests for the primitives
- Creating/modifying/deleting package namespaces or primitives
- Pushing/updating/removing builds/tests for the primitives
Notes
^^^^
- If "Person X" were to add "Person Y" in the namespace, "Person X" is the
reason why that message was created on the messaging bus and hence, only
"Person X" would be said to have done a "contribution activity" and while
"Person Y" was involved here, they would not be said to have done a
"contribution activity" in this case.
**Notes**
- If "Person X" were to add "Person Y" in the namespace, "Person X" is the reason
why that message was created on the messaging bus and hence, only "Person X" would
be said to have done a "contribution activity" and while "Person Y" was involved
here, they would not be said to have done a "contribution activity" in this case.
Ansible
----
-------
Application identifier - ``ansible``
Tasks
^^^^
**Tasks**
- Creating/modifying/deleting pull requests to the repository
- Making/updating/removing issue tickets on the repository
- Creating/modifying/deleting pull requests to the repository
- Making/updating/removing issue tickets on the repository
Notes
^^^^
- If "Person X" were to be assigned an issue ticket by "Person Y", "Person Y"
is the reason why that message was created on the messaging bus and hence,
only "Person Y" would be said to have done a "contribution activity" and not
"Person X" even when they were involved in this case.
**Notes**
- If "Person X" were to be assigned an issue ticket by "Person Y", "Person Y" is the
reason why that message was created on the messaging bus and hence, only "Person
Y" would be said to have done a "contribution activity" and not "Person X" even
when they were involved in this case.
Bodhi
----
-----
Application identifier - ``bodhi``
Tasks
^^^^
**Tasks**
- Creating/modifying/deleting new updates for the packages
- Giving karma to a testing package
- Creating/modifying/deleting new updates for the packages
- Giving karma to a testing package
Notes
^^^^
- If "Person X" were to give a negative karma to a release of a package that
belongs to "Person Y", "Person X" is the reason why that message was created
on the messaging bus and hence, only "Person X" would be said to have done
a "contribution activity" and while "Person Y" was involved here, they would
not be said to have done a "contribution activity" in this case.
**Notes**
- If "Person X" were to give a negative karma to a release of a package that belongs
to "Person Y", "Person X" is the reason why that message was created on the
messaging bus and hence, only "Person X" would be said to have done a
"contribution activity" and while "Person Y" was involved here, they would not be
said to have done a "contribution activity" in this case.
Discourse
----
---------
Application identifier - ``discourse``
Tasks
^^^^
**Tasks**
- Creating/modifying/deleting new threads on the forum
- Reacting to the existing threads
- Creating/modifying/deleting new threads on the forum
- Reacting to the existing threads
Notes
^^^^
- If "Person X" were to be tagged in a comment post made by "Person Y",
"Person Y" is the reason why that message was created on the messaging bus
and hence, only "Person Y" would be said to have done a "contribution
activity" and not "Person X" even when they were involved in this case.
**Notes**
- If "Person X" were to be tagged in a comment post made by "Person Y", "Person Y"
is the reason why that message was created on the messaging bus and hence, only
"Person Y" would be said to have done a "contribution activity" and not "Person X"
even when they were involved in this case.
Fedora Account System
----
---------------------
Application identifier - ``FAS``
Tasks
^^^^
**Tasks**
- Creating and modifying new accounts on the system
- Adding and removing accounts from groups
- Creating and modifying new accounts on the system
- Adding and removing accounts from groups
Notes
^^^^
- If "Person X" were to add "Person Y" to a FAS group - being the sponsor of
the same group, "Person X" is the reason why that message was created on the
messaging bus and hence, only "Person X" would be said to have done a
"contribution activity" and while "Person Y" was involved here, they would
not be said to have done a "contribution activity" in this case.
**Notes**
- If "Person X" were to add "Person Y" to a FAS group - being the sponsor of the
same group, "Person X" is the reason why that message was created on the messaging
bus and hence, only "Person X" would be said to have done a "contribution
activity" and while "Person Y" was involved here, they would not be said to have
done a "contribution activity" in this case.
Fedora Calendar
----
---------------
Application identifier - ``fedocal``
Tasks
^^^^
**Tasks**
- Creating/modifying/deleting of events on the calendar
- Transfer of events from one account to another
- Creating/modifying/deleting of events on the calendar
- Transfer of events from one account to another
Notes
^^^^
- If "Person X" were to be given the ownership of a calendar event by "Person
Y", "Person Y" is the reason why that message was created on the messaging
bus and hence, only "Person Y" would be said to have done a "contribution
activity" and not "Person X" even when they were involved in this case.
**Notes**
- If "Person X" were to be given the ownership of a calendar event by "Person Y",
"Person Y" is the reason why that message was created on the messaging bus and
hence, only "Person Y" would be said to have done a "contribution activity" and
not "Person X" even when they were involved in this case.
Fedora Build System
----
-------------------
Application identifier - ``koji``
Tasks
^^^^
**Tasks**
- Creating and deleting jobs for builds and testing
- Succeeding and failing of the aforementioned jobs
- Creating and deleting jobs for builds and testing
- Succeeding and failing of the aforementioned jobs
Notes
^^^^
- If "Person X" were to create a build job for a package owned by "Person Y",
"Person X" is the reason why that message was created on the messaging bus
and hence, only "Person X" would be said to have done a "contribution
activity" and while "Person Y" was involved here, they would not be said to
have done a "contribution activity" in this case.
**Notes**
- If "Person X" were to create a build job for a package owned by "Person Y",
"Person X" is the reason why that message was created on the messaging bus and
hence, only "Person X" would be said to have done a "contribution activity" and
while "Person Y" was involved here, they would not be said to have done a
"contribution activity" in this case.
Notifications
----
-------------
Application identifier - ``koji``
Tasks
^^^^
**Tasks**
- Creating/modifying/deleting rules from the dashboard
- Changing tracking rules and owners for an existing rule
- Creating/modifying/deleting rules from the dashboard
- Changing tracking rules and owners for an existing rule
Notes
^^^^
- If the activities done by "Person X" were to be tracked by "Person Y",
"Person Y" is the reason why that message was created on the messaging bus
and hence, only "Person Y" would be said to have done a "contribution
activity" and not "Person X" even when they were involved in this case.
**Notes**
- If the activities done by "Person X" were to be tracked by "Person Y", "Person Y"
is the reason why that message was created on the messaging bus and hence, only
"Person Y" would be said to have done a "contribution activity" and not "Person X"
even when they were involved in this case.
Pagure
----
------
Application identifier - ``pagure``
Tasks
^^^^
**Tasks**
- Creating/modifying/deleting repositories or groups
- Making pull requests against an existing repository
- Creating/modifying/deleting repositories or groups
- Making pull requests against an existing repository
Notes
^^^^
- If "Person X" were to make a pull request against a repository owned by
"Person Y", "Person X" is the reason why that message was created on the
messaging bus and hence, only "Person X" would be said to have done a
"contribution activity" and while "Person Y" was involved here, they
would not be said to have done a "contribution activity" in this case.
**Notes**
- If "Person X" were to make a pull request against a repository owned by "Person
Y", "Person X" is the reason why that message was created on the messaging bus and
hence, only "Person X" would be said to have done a "contribution activity" and
while "Person Y" was involved here, they would not be said to have done a
"contribution activity" in this case.
Elections
----
---------
Application identifier - ``elections``
Tasks
^^^^
**Tasks**
- Casting and changing vote for a preferred candidate during nomination
- Creating/modifying/deleting nominees from the stand
- Casting and changing vote for a preferred candidate during nomination
- Creating/modifying/deleting nominees from the stand
Notes
^^^^
- If "Person X" were to be voted for by "Person Y", "Person Y" is the reason
why that message was created on the messaging bus and hence, only "Person Y"
would be said to have done a "contribution activity" and not "Person X" even
when they were involved in this case.
**Notes**
- If "Person X" were to be voted for by "Person Y", "Person Y" is the reason why
that message was created on the messaging bus and hence, only "Person Y" would be
said to have done a "contribution activity" and not "Person X" even when they were
involved in this case.
Package Sources
----
---------------
Application identifier - ``distgit``
Tasks
^^^^
**Tasks**
- Creating/modifying/deleting repositories or groups
- Making/updating/removing pull requests on the repository
- Creating/modifying/deleting repositories or groups
- Making/updating/removing pull requests on the repository
Notes
^^^^
- If "Person X" were to be created a namespace for by a package sponsor
"Person Y", "Person X" is the reason why that message was created on the
messaging bus and hence, only "Person X" would be said to have done a
"contribution activity" and while "Person Y" was involved here, they
would not be said to have done a "contribution activity" in this case.
**Notes**
- If "Person X" were to be created a namespace for by a package sponsor "Person Y",
"Person X" is the reason why that message was created on the messaging bus and
hence, only "Person X" would be said to have done a "contribution activity" and
while "Person Y" was involved here, they would not be said to have done a
"contribution activity" in this case.

96
docs/fcas/solution_probntec.rst
View file

@ -1,62 +1,56 @@
.. _solution_probntec.rst:
Conundrum of Tracking of Non-Technical Contributions
====
====================================================
Problem statement
----
-----------------
Interactions with services are the sole means by which messages are published
on the `Fedora Messaging <https://fedora-messaging.readthedocs.io/>`_ bus,
resulting in their exclusive tracking. These services primarily cater to
technical contributions, thus posing a challenge in monitoring non-technical
contributions. Given the diverse nature of such contributions and the
subjective approaches employed to track them, devising an objective method to
describe them becomes notably arduous. What may be perceived as a contribution
by one individual may not be considered as such by another, thereby undermining
the service's ability to consistently track non-technical contributions.
Interactions with services are the sole means by which messages are published on the
`Fedora Messaging <https://fedora-messaging.readthedocs.io/>`_ bus, resulting in their
exclusive tracking. These services primarily cater to technical contributions, thus
posing a challenge in monitoring non-technical contributions. Given the diverse nature
of such contributions and the subjective approaches employed to track them, devising an
objective method to describe them becomes notably arduous. What may be perceived as a
contribution by one individual may not be considered as such by another, thereby
undermining the service's ability to consistently track non-technical contributions.
One potential workaround to address this issue involves implementing badges on
the `Fedora Badges <https://fedora-arc.readthedocs.io/en/latest/badges/index.html>`_
platform for various non-technical contributions. However, this approach
suffers from severe scalability issues when new badges need to be added
manually for each type of non-technical contribution. Consequently, it could
result in a scenario where there are thousands of badges associated with a
specific type of non-technical contribution, but only a few individuals
receiving each badge. Moreover, deviating from the standard practice of
assigning "subject" usernames as owners of contribution activities, adopting
Fedora Badges for contribution tracking would require both "subject" usernames
(the cause of the message publication event) and "object" usernames (involved
in the contribution activity) to possess ownership of a contribution activity.
One potential workaround to address this issue involves implementing badges on the
`Fedora Badges <https://fedora-arc.readthedocs.io/en/latest/badges/index.html>`_
platform for various non-technical contributions. However, this approach suffers from
severe scalability issues when new badges need to be added manually for each type of
non-technical contribution. Consequently, it could result in a scenario where there are
thousands of badges associated with a specific type of non-technical contribution, but
only a few individuals receiving each badge. Moreover, deviating from the standard
practice of assigning "subject" usernames as owners of contribution activities, adopting
Fedora Badges for contribution tracking would require both "subject" usernames (the
cause of the message publication event) and "object" usernames (involved in the
contribution activity) to possess ownership of a contribution activity.
In the context of a contribution event, let us consider a scenario involving
"Member X," who possesses the authority to bestow "Badge A" upon individuals
who successfully complete "Task B." In this case, "Member X" awards "Badge A" to
"Member Y." However, due to the non-technical nature of this contribution, the
Fedora Badges' Message Consumer entity fails to automatically track the
occurrence of "Member Y" performing "Task B" with reliability. The act of
awarding the badge serves as confirmation for two distinct contribution
activities. The first activity involves "Member Y" carrying out "Task B," while
the second activity entails "Member X" granting "Badge A" to "Member Y." In
essence, "Member Y" is responsible for the primary contribution, but they
depend on "Member X" to receive the badge for their contribution to be
accurately recorded. On the other hand, the act of awarding the badge itself
constitutes a separate contribution, making "Member X" the owner of this
secondary activity.
In the context of a contribution event, let us consider a scenario involving "Member X,"
who possesses the authority to bestow "Badge A" upon individuals who successfully
complete "Task B." In this case, "Member X" awards "Badge A" to "Member Y." However, due
to the non-technical nature of this contribution, the Fedora Badges' Message Consumer
entity fails to automatically track the occurrence of "Member Y" performing "Task B"
with reliability. The act of awarding the badge serves as confirmation for two distinct
contribution activities. The first activity involves "Member Y" carrying out "Task B,"
while the second activity entails "Member X" granting "Badge A" to "Member Y." In
essence, "Member Y" is responsible for the primary contribution, but they depend on
"Member X" to receive the badge for their contribution to be accurately recorded. On the
other hand, the act of awarding the badge itself constitutes a separate contribution,
making "Member X" the owner of this secondary activity.
Probable workaround
----
-------------------
This method should only be employed as a final recourse for monitoring
non-technical contributions that do not automatically generate a message on
the Fedora Messaging bus upon occurrence. Since the badges cannot be closely
linked to the contribution activity itself and the tracking process is manual
in nature, they are highly susceptible to being left untracked. This can happen
if the contribution owner forgets to claim a badge when they perform the
contribution activity, or if the badge facilitator is unavailable and causes
delays in tracking the activity. Additionally, there is a risk of unfair
tracking, such as the badge facilitator granting a specific badge to their
friends who may not have actually completed the corresponding contribution
activity, or random individuals claiming the badge through the associated link
in bad faith. In the broader context, this will significantly skew the
contributor statistics in a negative manner.
This method should only be employed as a final recourse for monitoring non-technical
contributions that do not automatically generate a message on the Fedora Messaging bus
upon occurrence. Since the badges cannot be closely linked to the contribution activity
itself and the tracking process is manual in nature, they are highly susceptible to
being left untracked. This can happen if the contribution owner forgets to claim a badge
when they perform the contribution activity, or if the badge facilitator is unavailable
and causes delays in tracking the activity. Additionally, there is a risk of unfair
tracking, such as the badge facilitator granting a specific badge to their friends who
may not have actually completed the corresponding contribution activity, or random
individuals claiming the badge through the associated link in bad faith. In the broader
context, this will significantly skew the contributor statistics in a negative manner.

14
docs/fcas/solution_techtool.rst
View file

@ -1,15 +1,15 @@
.. _solution_techtool.rst:
Technologies suggested
====
======================
Keeping in mind that the project will potentially contribute to the maintenance
overhead of the team, most of the technologies suggested below inherit from the
team's current expertise and a divergence from the same into using newer and
lesser known technologies have been made only when it was absolutely warranted.
Keeping in mind that the project will potentially contribute to the maintenance overhead
of the team, most of the technologies suggested below inherit from the team's current
expertise and a divergence from the same into using newer and lesser known technologies
have been made only when it was absolutely warranted.
External technologies
----
---------------------
1. **Visualization:** `Grafana <https://grafana.com/>`_
2. **Database:** `Postgres <https://www.postgresql.org/>`_
@ -17,7 +17,7 @@ External technologies
4. **Programming language:** `Python 3 <https://www.python.org/downloads/>`_
In-house technologies
----
---------------------
1. `Fedora Messaging <https://fedora-messaging.readthedocs.io/>`_
2. `Datagrepper <https://apps.fedoraproject.org/datagrepper/>`_

23
docs/flask-oidc/authlib_code.rst
View file

@ -1,22 +1,21 @@
Authlib
=========
=======
Discussion
----------
Authlib implementation for Flask. Flask-oidc uses ouath2client to create a flow
based on client secrets using this function `client.flow_from_clientsecrets https://oauth2client.readthedocs.io/en/latest/source/oauth2client.client.html#oauth2client.client.flow_from_clientsecrets`_.
to create an ouath2 webserver flow which validates based on oidc tokens.
This needs to be replaced using a new library.
Authlib implementation for Flask. Flask-oidc uses ouath2client to create a flow based on
client secrets using this function `client.flow_from_clientsecrets
<https://oauth2client.readthedocs.io/en/latest/source/oauth2client.client.html#oauth2client.client.flow_from_clientsecrets>`_.
to create an ouath2 webserver flow which validates based on oidc tokens. This needs to
be replaced using a new library.
Authlib
-------
Authlib has the functionality to follow this flow and use oidc on top of oauth2
functionality https://docs.authlib.org/en/latest/client/oauth2.html#oauth-2-openid-connect
functionality
https://docs.authlib.org/en/latest/client/oauth2.html#oauth-2-openid-connect
Conclusion
----------
Authlib provides the necessary tooling to do this replacement which shows it as
suitable for a replacement
Authlib provides the necessary tooling to do this replacement which shows it as suitable
for a replacement

22
docs/flask-oidc/index.rst
View file

@ -4,16 +4,17 @@ Flask-oidc
Purpose
-------
Flask-oidc is a library used across the Fedora infrastructure. For its
authentication it the oauth2client library is used. This is now deprecated and
no longer maintained. This will need to be replaced.
Flask-oidc is a library used across the Fedora infrastructure. For its authentication it
the oauth2client library is used. This is now deprecated and no longer maintained. This
will need to be replaced.
Resources
---------
* Flask-oidc code: https://github.com/puiterwijk/flask-oidc
* Bodhi implementation: https://github.com/fedora-infra/bodhi/tree/develop/bodhi-server/bodhi/server/auth
* Authlib docs: https://docs.authlib.org/en/latest/basic/index.html
- Flask-oidc code: https://github.com/puiterwijk/flask-oidc
- Bodhi implementation:
https://github.com/fedora-infra/bodhi/tree/develop/bodhi-server/bodhi/server/auth
- Authlib docs: https://docs.authlib.org/en/latest/basic/index.html
Investigation
-------------
@ -26,7 +27,6 @@ Here is the list of ideas/things we discussed/looked at:
oauthlib_authlib
authlib_code
Conclusions
-----------
@ -34,11 +34,9 @@ Authlib is the new industry standard and looks to be a suitable replacement for
oauth2client. The flask-oidc code would need to be rewritten to replace all the
oauth2client code.
The flow of the code would need to be adjusted to work with the new library.
Google-auth and authlib were looked at for this investigation and it was
suggested that authlib looks a better replacemnt.
The flow of the code would need to be adjusted to work with the new library. Google-auth
and authlib were looked at for this investigation and it was suggested that authlib
looks a better replacemnt.
Proposed Roadmap
----------------

24
docs/flask-oidc/oauthlib_authlib.rst
View file

@ -4,23 +4,23 @@ Oauthlib vs Authlib
Resources
---------
* Oauthlib: https://oauthlib.readthedocs.io/en/latest/
* Authlib: https://docs.authlib.org/en/latest/
- Oauthlib: https://oauthlib.readthedocs.io/en/latest/
- Authlib: https://docs.authlib.org/en/latest/
Discussion
----------
Which library should be used to replace the currently used and deprecated
oauth2client. The two main contenders are Oauthlib and Authlib.
Authlib is the more popular of the two choices as seen in this `graph https://python.libhunt.com/compare-authlib-vs-oauthlib`_.
For flask the oauthlib developers themselves recommend using authilib
Which library should be used to replace the currently used and deprecated oauth2client.
The two main contenders are Oauthlib and Authlib.
Authlib is the more popular of the two choices as seen in this `graph
<https://python.libhunt.com/compare-authlib-vs-oauthlib>`_. For flask the oauthlib
developers themselves recommend using authilib
https://github.com/lepture/flask-oauthlib#readme
Conclusion
----------
Authlib seems to be the best candidate for this initiative. There is knowledge
in the team of the library as it was used in bodhi. It also appears to have an
easier to implement and has a flask integration and is recommended for use by
the developer.
Authlib seems to be the best candidate for this initiative. There is knowledge in the
team of the library as it was used in bodhi. It also appears to have an easier to
implement and has a flask integration and is recommended for use by the developer.

523
docs/fmn/april2022/index.rst
View file

@ -1,66 +1,68 @@
FMN Replacement (April 2022)
============================
FMN is a web application allowing users to create filters on messages sent to fedmsg
and forward these as notifications on to email or IRC
FMN is a web application allowing users to create filters on messages sent to fedmsg and
forward these as notifications on to email or IRC
* `current deployment <https://apps.fedoraproject.org/notifications/>`__
* `FMN source <https://github.com/fedora-infra/fmn>`__
* `CPE initiative ticket <https://pagure.io/cpe/initiatives-proposal/issue/10>`__
- `current deployment <https://apps.fedoraproject.org/notifications/>`__
- `FMN source <https://github.com/fedora-infra/fmn>`__
- `CPE initiative ticket <https://pagure.io/cpe/initiatives-proposal/issue/10>`__
Issues to solve
---------------
Message Delivery Lag
~~~~~~~~~~~~~~~~~~~~
In times of peak congestion on the message queue (e.g. during a mass rebuild) it can take **days** for messages to be delivered to users.
In times of peak congestion on the message queue (e.g. during a mass rebuild) it can
take **days** for messages to be delivered to users.
fedmsg
~~~~~~
The current version of FMN was implemented uses the deprecated fedmsg bus, not the currently supported Fedora Messaging (rabbitmq) message queue.
Additionally, the FMN code is deeply integrated with the fedmsg ecosystem (e.g. the configuration setup for FMN is kept in the fedmsg.d/ directory)
The current version of FMN was implemented uses the deprecated fedmsg bus, not the
currently supported Fedora Messaging (rabbitmq) message queue.
Additionally, the FMN code is deeply integrated with the fedmsg ecosystem (e.g. the
configuration setup for FMN is kept in the fedmsg.d/ directory)
UI
---
--
After adhoc discussions with the development community one of the major issues is the FMN web UI:
* The interface for creating new filters is complicated and confusing.
* Users have to define filters seperately for different destentations (i.e. email and IRC) This is due to how the database is designed.
After adhoc discussions with the development community one of the major issues is the
FMN web UI:
- The interface for creating new filters is complicated and confusing.
- Users have to define filters seperately for different destentations (i.e. email and
IRC) This is due to how the database is designed.
Design options
--------------
- jcline’s option: write a RabbitMQ extension. While it would probably
be way more efficient we don’t have the knowledge to write or
maintain this code. Also, it would require adding headers for the
fields we want to filter on, such as usernames and packages, and
we’ll probably loose in flexibility.
- write a fedora-messaging consumer that would triage incoming message
and produce add notifications to one queue per destination (email,
irc, matrix). They would be on a FMN-specific vhost on the RabbitMQ
server. Then write AMQP consumers for these queues to actually send
the notifications. This would allow the IRC and Matrix notifiers to
maintain a permanent connection to the IRC/Matrix server(s).
- jcline’s option: write a RabbitMQ extension. While it would probably be way more
efficient we don’t have the knowledge to write or maintain this code. Also, it would
require adding headers for the fields we want to filter on, such as usernames and
packages, and we’ll probably loose in flexibility.
- write a fedora-messaging consumer that would triage incoming message and produce add
notifications to one queue per destination (email, irc, matrix). They would be on a
FMN-specific vhost on the RabbitMQ server. Then write AMQP consumers for these queues
to actually send the notifications. This would allow the IRC and Matrix notifiers to
maintain a permanent connection to the IRC/Matrix server(s).
This work requires all applications to use Fedora Messaging Message
Schemas, because the triage component will rely on schemas to extract
information from the messages.
This work requires all applications to use Fedora Messaging Message Schemas, because the
triage component will rely on schemas to extract information from the messages.
Later on, when we refer to artifacts, we mean artifacts produced by the
Fedora infrastructure: packages, containers, modules, flatpaks.
Later on, when we refer to artifacts, we mean artifacts produced by the Fedora
infrastructure: packages, containers, modules, flatpaks.
Stakeholders
------------
- `@devel list question to gather
requirements <https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/SGGM3XPGQX77SEDOLEI376SZVRFKTXGR/>`__
- `Fedora Discussion
thread <https://discussion.fedoraproject.org/t/notifications-application-aka-fmn-aka-apps-fedoraproject-org-notifications/36914/2>`__
- Also ask the @mindshare / @ambassadors for the needs of non-packagers
- `@devel list question to gather requirements
<https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/SGGM3XPGQX77SEDOLEI376SZVRFKTXGR/>`__
- `Fedora Discussion thread
<https://discussion.fedoraproject.org/t/notifications-application-aka-fmn-aka-apps-fedoraproject-org-notifications/36914/2>`__
- Also ask the @mindshare / @ambassadors for the needs of non-packagers
Requirements
------------
@ -68,204 +70,182 @@ Requirements
Tracking
~~~~~~~~
- As a Fedora Contributor, I want to ask to be notified of what happens
on artifacts I own
- As a Fedora Contributor, I want to ask to be notified of what happens
on artifacts owned by a group I am a member of
- As a Fedora User, I want to ask to be notified of what happens on any
artifact by entering its type and its name
- As a Fedora User, I want to be notified of events referring to my
username
- As a Fedora User, I want to be notified of events referring to a
group I am a member of (requires a `new
feature <https://github.com/fedora-infra/fedora-messaging/issues/252>`__
in Fedora Messaging)
- As a Fedora User, I want to select which groups I want to be notified
about among the groups I’m a member of (some groups own so many
artifacts it would be too many notifications, eg the python
maintainers group)
- As a Fedora User, I want to be able to follow someone (for example,
my mentee)
- As a Fedora Contributor, I want to ask to be notified of what happens on artifacts I
own
- As a Fedora Contributor, I want to ask to be notified of what happens on artifacts
owned by a group I am a member of
- As a Fedora User, I want to ask to be notified of what happens on any artifact by
entering its type and its name
- As a Fedora User, I want to be notified of events referring to my username
- As a Fedora User, I want to be notified of events referring to a group I am a member
of (requires a `new feature
<https://github.com/fedora-infra/fedora-messaging/issues/252>`__ in Fedora Messaging)
- As a Fedora User, I want to select which groups I want to be notified about among the
groups I’m a member of (some groups own so many artifacts it would be too many
notifications, eg the python maintainers group)
- As a Fedora User, I want to be able to follow someone (for example, my mentee)
Filtering
~~~~~~~~~
- As a Fedora User, I want to be able to set the minimum severity level
on each rule (Fedora Messages have the notion of severity) (optional)
- As a Fedora User, I want to be able to block or allow notifications
from a particular application (koji, bodhi, dist-git, etc)
- As a Fedora User, I want to be able to disable notifications for my
own actions. They should be disabled by default.
- As a Fedora User, I want to be able to block or allow notifications
from a particular topic. The list of known topics can be obtained
from the installed schemas. This overrides the app allow/block list,
for example I don’t want koji messages except for the “build
complete” ones.
- As a Fedora User, I want to be able to set the minimum severity level on each rule
(Fedora Messages have the notion of severity) (optional)
- As a Fedora User, I want to be able to block or allow notifications from a particular
application (koji, bodhi, dist-git, etc)
- As a Fedora User, I want to be able to disable notifications for my own actions. They
should be disabled by default.
- As a Fedora User, I want to be able to block or allow notifications from a particular
topic. The list of known topics can be obtained from the installed schemas. This
overrides the app allow/block list, for example I don’t want koji messages except for
the “build complete” ones.
Destinations
~~~~~~~~~~~~
- As a Fedora User, I want to my notifications to be sent via email if
I set so.
- As a Fedora User, I want to my notifications to be sent via IRC if I
set so.
- As a Fedora User, I want to my notifications to be sent via Matrix if
I set so (optional).
- As a Fedora User, I want to be able to choose one or more
destination(s) on each rule (my objects, object foo, object bar, etc)
- As a Fedora user, I want my address for each destination to be
retreived from my FAS account. If I have multiple values set for a
destination, I want to be able to choose one.
- As a Fedora User, I want to be able to do filtering on my
notification email based on application name and/or topic. A good way
to offer this possibility would be to set these values as email
headers
- As a Fedora User, I want to my notifications to be sent via email if I set so.
- As a Fedora User, I want to my notifications to be sent via IRC if I set so.
- As a Fedora User, I want to my notifications to be sent via Matrix if I set so
(optional).
- As a Fedora User, I want to be able to choose one or more destination(s) on each rule
(my objects, object foo, object bar, etc)
- As a Fedora user, I want my address for each destination to be retreived from my FAS
account. If I have multiple values set for a destination, I want to be able to choose
one.
- As a Fedora User, I want to be able to do filtering on my notification email based on
application name and/or topic. A good way to offer this possibility would be to set
these values as email headers
Deployment
~~~~~~~~~~
- As a Fedora Sysadmin, I want to be able to run multiple instance of
any component of FMN in Openshift if I need to, for performance
reasons.
- As a Fedora User, I want only one notification sent when a message
matches my filters. Do note that during schema transition periods,
the same event may produce multiple messages on different topics
(``fedora.event.v1`` and ``fedora.event.v2``). I think it would be
preferrable to subscribe the triage component to a fixed list of
known topics instead of subscribing it to everything (``#``).
- As a Fedora Sysadmin, I want the Destination Handler components to be
as simple and fast as possible because it will not be easy to have
multiple instances of them (not sure if IRC allows multiple
connections of the same account)
- As a Fedora Sysadmin, I want to be able to run multiple instance of any component of
FMN in Openshift if I need to, for performance reasons.
- As a Fedora User, I want only one notification sent when a message matches my filters.
Do note that during schema transition periods, the same event may produce multiple
messages on different topics (``fedora.event.v1`` and ``fedora.event.v2``). I think it
would be preferrable to subscribe the triage component to a fixed list of known topics
instead of subscribing it to everything (``#``).
- As a Fedora Sysadmin, I want the Destination Handler components to be as simple and
fast as possible because it will not be easy to have multiple instances of them (not
sure if IRC allows multiple connections of the same account)
Misc
~~~~
- As an Application Developper, I want to set the summary and the body
(where applicable) of what will be sent out to users in my message
schema.
- As a Fedora User, I want to set my notification rules via a web UI
- As an Application Developper, I want to set the summary and the body (where
applicable) of what will be sent out to users in my message schema.
- As a Fedora User, I want to set my notification rules via a web UI
Suggested design
----------------
.. image:: workflow.png
Data model and configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- There are multiple Tracking Rules defined in FMN, that accept zero or
more parameters.
- There are multiple supported Destinations in FMN
- There are multiple Filters defined in FMN, that accept zero or more
parameters.
- Users can activate zero or more Tracking Rules and set the
corresponding parameters if applicable.
- Users can set one or more Destinations for a Tracking Rule
- Users can set one or more Filters for a Tracking Rule and a
Destination, and set the corresponding parameters if applicable.
- There are multiple Tracking Rules defined in FMN, that accept zero or more parameters.
- There are multiple supported Destinations in FMN
- There are multiple Filters defined in FMN, that accept zero or more parameters.
- Users can activate zero or more Tracking Rules and set the corresponding parameters if
applicable.
- Users can set one or more Destinations for a Tracking Rule
- Users can set one or more Filters for a Tracking Rule and a Destination, and set the
corresponding parameters if applicable.
Workflow
~~~~~~~~
- A Runner is created for each Tracking Rule
- Incoming messages are passed to each Runner (parallelism is possible
here)
- The Runner applies the Tracking Rule over the messages, which
produces a list of usernames to notify
- For each username:
- A Runner is created for each Tracking Rule
- Incoming messages are passed to each Runner (parallelism is possible here)
- The Runner applies the Tracking Rule over the messages, which produces a list of
usernames to notify
- For each username:
- check which destination(s) they want for this Tracking Rule
- For each destination:
- check which destination(s) they want for this Tracking Rule
- For each destination:
- check if the incoming message’s properties pass all the Filters
for this Tracking Rule and this Destination
- If so, lookup the desired address in the DB
- check that the value is still in the user’s FAS profile, or
fallback to the FAS profile’s first value
- Produce a Notification for this incoming message. The
Notification will contain the destination name, the address, a
subject (optional), and a body.
- check if the incoming message’s properties pass all the Filters for this Tracking
Rule and this Destination
- If so, lookup the desired address in the DB
- check that the value is still in the user’s FAS profile, or fallback to the FAS
profile’s first value
- Produce a Notification for this incoming message. The Notification will contain
the destination name, the address, a subject (optional), and a body.
- Notifications produced by all the Tracking Rules are de-duplicated
before being added to the Destination queues. One incoming message
should produce at most one Notification for a user and a Destination.
- Destination Handlers are separate processes that will get
notifications from their queue and send the notification to the
corresponding service.
- Notifications produced by all the Tracking Rules are de-duplicated before being added
to the Destination queues. One incoming message should produce at most one
Notification for a user and a Destination.
- Destination Handlers are separate processes that will get notifications from their
queue and send the notification to the corresponding service.
Possible tracking rules
~~~~~~~~~~~~~~~~~~~~~~~
Artifact owner
^^^^^^^^^^^^^^
++++++++++++++
For each artifact in the message, get the artifact owners. For each
owner, check in the DB whether they have selected to be notified for
their artifacts. If so, add a notification for each selected destination
if they pass filtering.
For each artifact in the message, get the artifact owners. For each owner, check in the
DB whether they have selected to be notified for their artifacts. If so, add a
notification for each selected destination if they pass filtering.
Artifact owned by one of my selected groups
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+++++++++++++++++++++++++++++++++++++++++++
For each artifact in the message, get the artifact owner groups. For
each groups, get the list of members and check in the DB whether they
have selected to be notified for artifacts in this group. If so, add a
notification for each selected destination if they pass filtering.
For each artifact in the message, get the artifact owner groups. For each groups, get
the list of members and check in the DB whether they have selected to be notified for
artifacts in this group. If so, add a notification for each selected destination if they
pass filtering.
Artifact follower
^^^^^^^^^^^^^^^^^
+++++++++++++++++
For each artifact in the message, get in the DB the list of users who
follow those artifacts. For each of them, add a notification for each
selected destination if they pass filtering.
For each artifact in the message, get in the DB the list of users who follow those
artifacts. For each of them, add a notification for each selected destination if they
pass filtering.
Events referring to me
^^^^^^^^^^^^^^^^^^^^^^
++++++++++++++++++++++
For each username in the message, check in the DB whether they have
selected to be notified for events referring to them. If so, add a
notification for each selected destination if they pass filtering.
For each username in the message, check in the DB whether they have selected to be
notified for events referring to them. If so, add a notification for each selected
destination if they pass filtering.
Followers
^^^^^^^^^
+++++++++
For each username in the message, get in the DB the list of users who
follow them, and add a notification for each selected destination if
they pass filtering.
For each username in the message, get in the DB the list of users who follow them, and
add a notification for each selected destination if they pass filtering.
Possible Filters
~~~~~~~~~~~~~~~~
The Filters should take the destination into account, to be able to
filter a noisy app on IRC but not on email for example.
The Filters should take the destination into account, to be able to filter a noisy app
on IRC but not on email for example.
Application
^^^^^^^^^^^
+++++++++++
If the application is in the user’s blocklist for the destination, and
not in the user’s allowlist: cancel the notification. The list of
supported applications to present in the UI can be extracted from the
installed message schemas.
If the application is in the user’s blocklist for the destination, and not in the user’s
allowlist: cancel the notification. The list of supported applications to present in the
UI can be extracted from the installed message schemas.
Severity
^^^^^^^^
++++++++
If the severity is below the minimal severity threshold that the user
has set for this destination, cancel the notification.
If the severity is below the minimal severity threshold that the user has set for this
destination, cancel the notification.
My actions
^^^^^^^^^^
++++++++++
If the message results of this user’s action, cancel the notification.
Topic
^^^^^
+++++
If the topic matches (*fnmatch*) any of the user’s topic blocklist
entries, and does not match any of the user’s topic allowlist entries:
cancel the notification
If the topic matches (*fnmatch*) any of the user’s topic blocklist entries, and does not
match any of the user’s topic allowlist entries: cancel the notification
Existing FMN
------------
@ -273,180 +253,187 @@ Existing FMN
Current FMN code and usage analysis
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The current FMN application implements a fedmsg based consumer, and a
Flask based webui for users to create, modify, and manage their filters.
The current FMN application implements a fedmsg based consumer, and a Flask based webui
for users to create, modify, and manage their filters.
Current Users
^^^^^^^^^^^^^
+++++++++++++
Currently there are 6,777 users with accounts on FMN.
Contexts
^^^^^^^^
++++++++
A context in FMN is the top-level for a user to define their filters. A
context is where a message is sent after being matched. FMN in
production has 2 contexts that users can use:
A context in FMN is the top-level for a user to define their filters. A context is where
a message is sent after being matched. FMN in production has 2 contexts that users can
use:
- email
- irc
- email
- irc
Each of these contexts can be enabled, which disables all the filters
defined under that context.
Each of these contexts can be enabled, which disables all the filters defined under that
context.
Of the 6,777 users in FMN, the following is a breakdown of these users’
enabled contexts:
Of the 6,777 users in FMN, the following is a breakdown of these users’ enabled
contexts:
- Both email and irc enabled: **1048 users**
- Only email enabled: **4619 users**
- Only irc enabled: **169 users**
- No contexts enabled: **941 users**
- Both email and irc enabled: **1048 users**
- Only email enabled: **4619 users**
- Only irc enabled: **169 users**
- No contexts enabled: **941 users**
Filters and Rules
^^^^^^^^^^^^^^^^^
+++++++++++++++++
For each context, a user defines zero to many filters. Each filter is
made up of 1 to many different rules which are defined in code.
For each context, a user defines zero to many filters. Each filter is made up of 1 to
many different rules which are defined in code.
Filters defaults
''''''''''''''''
................
When a user logs into FMN for the first time, they are set up with a
copy of the default filters.
When a user logs into FMN for the first time, they are set up with a copy of the default
filters.
The current set of default filters and rules are:
- Events on packages that I own’:
- Events on packages that I own’:
- [‘fmn.rules:user_package_filter’, {‘fasnick’: username}],
- [‘fmn.rules:user_filter’, {‘fasnick’: username}],
- [‘fmn.rules:koji_instance’, {‘instance’: ‘ppc,s390,arm’}],
- [‘fmn.rules:taskotron_result_new’, {}],
- [‘fmn.rules:git_pkgdb2branch_start’, {}],
- [‘fmn.rules:git_pkgdb2branch_complete’, {}],
- [‘fmn.rules:fedoratagger_catchall’, {}],
- [‘fmn.rules:faf_report_threshold1’, {}],
- [‘fmn.rules:faf_problem_threshold1’, {}],
- [‘fmn.rules:bodhi_masher_start’, {}],
- [‘fmn.rules:mdapi_repo_update’, {}],
- [‘fmn.rules:ci_step_complete’, {}],
- [‘fmn.rules:copr_build_start’, {}],
- [‘fmn.rules:copr_chroot_start’, {}],
- [‘fmn.rules:copr_worker_create’, {}],
- [‘fmn.rules:koji_scratch_build_started’, {}],
- [‘fmn.rules:koji_build_started’, {}],
- [‘fmn.rules:koji_rpm_sign’, {}],
- [‘fmn.rules:koji_tag’, {}],
- [‘fmn.rules:koji_untag’, {}],
- [‘fmn.rules:koji_repo_done’, {}],
- [‘fmn.rules:koji_repo_init’, {}]
- [‘fmn.rules:koji_package_list_change’, {}]
- [‘fmn.rules:summershum_catchall’, {}]
- [‘fmn.rules:anitya_catchall’, {}]]
- [‘fmn.rules:user_package_filter’, {‘fasnick’: username}],
- [‘fmn.rules:user_filter’, {‘fasnick’: username}],
- [‘fmn.rules:koji_instance’, {‘instance’: ‘ppc,s390,arm’}],
- [‘fmn.rules:taskotron_result_new’, {}],
- [‘fmn.rules:git_pkgdb2branch_start’, {}],
- [‘fmn.rules:git_pkgdb2branch_complete’, {}],
- [‘fmn.rules:fedoratagger_catchall’, {}],
- [‘fmn.rules:faf_report_threshold1’, {}],
- [‘fmn.rules:faf_problem_threshold1’, {}],
- [‘fmn.rules:bodhi_masher_start’, {}],
- [‘fmn.rules:mdapi_repo_update’, {}],
- [‘fmn.rules:ci_step_complete’, {}],
- [‘fmn.rules:copr_build_start’, {}],
- [‘fmn.rules:copr_chroot_start’, {}],
- [‘fmn.rules:copr_worker_create’, {}],
- [‘fmn.rules:koji_scratch_build_started’, {}],
- [‘fmn.rules:koji_build_started’, {}],
- [‘fmn.rules:koji_rpm_sign’, {}],
- [‘fmn.rules:koji_tag’, {}],
- [‘fmn.rules:koji_untag’, {}],
- [‘fmn.rules:koji_repo_done’, {}],
- [‘fmn.rules:koji_repo_init’, {}]
- [‘fmn.rules:koji_package_list_change’, {}]
- [‘fmn.rules:summershum_catchall’, {}]
- [‘fmn.rules:anitya_catchall’, {}]]
- ‘Events referring to my username’: [[‘fmn.rules:user_filter’,
{‘fasnick’: username}],]
- ‘Critical taskotron tasks on my packages’
- ‘Events referring to my username’: [[‘fmn.rules:user_filter’, {‘fasnick’: username}],]
- ‘Critical taskotron tasks on my packages’
- [‘fmn.rules:user_package_filter’, {‘fasnick’: username}],
- [‘fmn.rules:taskotron_release_critical_task’, {}],
- [‘fmn.rules:taskotron_task_particular_or_changed_outcome’,
{‘outcome’: ‘FAILED’}]
- [‘fmn.rules:user_package_filter’, {‘fasnick’: username}],
- [‘fmn.rules:taskotron_release_critical_task’, {}],
- [‘fmn.rules:taskotron_task_particular_or_changed_outcome’, {‘outcome’: ‘FAILED’}]
Because the configuration of the default setup has changed over the
years, users that logged in earlier have older default filter setups,
making it harder to figure out who is using defaults.
Because the configuration of the default setup has changed over the years, users that
logged in earlier have older default filter setups, making it harder to figure out who
is using defaults.
Current UI
^^^^^^^^^^
Login page
''''''''''
::
++++++++++
apps.fedoraproject.org/notifications/
Login page
..........
.. code-block::
apps.fedoraproject.org/notifications/
.. image:: oldscreenshots/login.png
About page
''''''''''
::
..........
apps.fedoraproject.org/notifications/about
.. code-block::
apps.fedoraproject.org/notifications/about
.. image:: oldscreenshots/about.png
Profile page
''''''''''''
::
............
apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/
.. code-block::
apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/
.. image:: oldscreenshots/profile.png
Filter list per context
'''''''''''''''''''''''
::
.......................
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/irc/
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/email/
.. code-block::
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/irc/
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/email/
.. image:: oldscreenshots/filterlist.png
Create / Edit Filter page
'''''''''''''''''''''''''
::
.........................
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/irc/<filter_id>/
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/email/<filter_id>/
.. code-block::
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/irc/<filter_id>/
https://apps.fedoraproject.org/notifications/<fedora_username>.id.fedoraproject.org/email/<filter_id>/
.. image:: oldscreenshots/createfilter.png
Skillsets
---------
* fedora messaging
* Web application UI
* Python (database)
* Rabbitmq (used in the backend of FMN and could be useful to know)
* Openshift
* IRC - connectivity to new systems in use (matrix)
- fedora messaging
- Web application UI
- Python (database)
- Rabbitmq (used in the backend of FMN and could be useful to know)
- Openshift
- IRC - connectivity to new systems in use (matrix)
Open Questions
--------------
* What are the core services the new FMN will need to support?
* Are there fedora-messaging schemas for each of the core services in place already?
* Does the schemas need revision, ie do they still work?
* Who would be good to get a review on UI? Mo Duffy potentially? Could this go in as a blog post of sorts?
- What are the core services the new FMN will need to support?
- Are there fedora-messaging schemas for each of the core services in place already?
- Does the schemas need revision, ie do they still work?
- Who would be good to get a review on UI? Mo Duffy potentially? Could this go in as a
blog post of sorts?
Possible Timeline
-----------------
In 1 month
~~~~~~~~~~
* Email notifying community about work started on new FMN service has gone out
* The development team has been onboarded
* The backlog has epic level tickets created
* Testing criteria is loosely defined for performance, ie what is the min/max acceptable time for receiving a notification
- Email notifying community about work started on new FMN service has gone out
- The development team has been onboarded
- The backlog has epic level tickets created
- Testing criteria is loosely defined for performance, ie what is the min/max acceptable
time for receiving a notification
In 3 months
~~~~~~~~~~~
* An update is sent to the lists on work underway
* Infra leads have been invited to a review of the work to date
* A firmer timeline for project delivery has been defined
* There is testing criteria/benchmarking agreed to
- An update is sent to the lists on work underway
- Infra leads have been invited to a review of the work to date
- A firmer timeline for project delivery has been defined
- There is testing criteria/benchmarking agreed to
In 6 months
~~~~~~~~~~~
* The service is deployed in staging *or* its in production but it is not enabled for everyone yet
* Testers are invited to onboard into the new service and help test performance
* An outage period for cutover to the new service has been identified and planned if needed
- The service is deployed in staging *or* its in production but it is not enabled for
everyone yet
- Testers are invited to onboard into the new service and help test performance
- An outage period for cutover to the new service has been identified and planned if
needed
In 9 months
~~~~~~~~~~~
* Hotfixes are applied
- Hotfixes are applied

2
docs/fmn/index.rst
View file

@ -1,5 +1,5 @@
FMN
===========
===
.. toctree::
:maxdepth: 3

47
docs/fmn/june2021/index.rst
View file

@ -1,8 +1,9 @@
FMN replace or fix research (June 2021)
=======================================
`FMN <https://apps.fedoraproject.org/notifications/>`_ is application where users with fedora account
setup notifications to be sent on fedora-messaging messages that match a specific filter.
`FMN <https://apps.fedoraproject.org/notifications/>`_ is application where users with
fedora account setup notifications to be sent on fedora-messaging messages that match a
specific filter.
Our current deployment running on https://apps.fedoraproject.org/notifications/.
@ -13,15 +14,15 @@ The software is currently under our fedora-infra organization.
Abstract take-away
------------------
FMN is a project that allows people in our community to get notified when messages that interest them
fire on the message-bus, making the message-bus more useful to people that are not
directly developing or troubleshooting applications running in our infra.
FMN is a project that allows people in our community to get notified when messages that
interest them fire on the message-bus, making the message-bus more useful to people that
are not directly developing or troubleshooting applications running in our infra.
This means we really don't want to drop it's support.
A two person team with skill in python development, with access to current FMN infrastructure,
and ability to run deployment playbooks should be able to finish
the work started in Pierre's POC and end up with up-to-date and more maintainable project.
A two person team with skill in python development, with access to current FMN
infrastructure, and ability to run deployment playbooks should be able to finish the
work started in Pierre's POC and end up with up-to-date and more maintainable project.
Solutions to be explored
------------------------
@ -30,41 +31,43 @@ We have three options:
- Upgrade FMN to supported version of Python and invest in the maintenance
- Create a new bespoke application and invest in the maintenanece
- Find off-the shelf application that would fill the role without us needing to maintain it
- Find off-the shelf application that would fill the role without us needing to maintain
it
Currently we are proposing primarily the first option.
Preliminary notes on maintaining current FMN
--------------------------------------------
Based on Pierre's POC, FMN is upgrade-able to recent versions
of python and its libraries.
Based on Pierre's POC, FMN is upgrade-able to recent versions of python and its
libraries.
This suggests it is prudent to invest in it further
- FMN is key piece to fullfil the promise of having a message-bus useful to our community
- FMN is key piece to fullfil the promise of having a message-bus useful to our
community
- green-field development is more costly and harder to ramp up
Outstanding issues to be tackled by the initiative:
- verify the upgrades actually work by thorough testing
- migrate FMN to the fedora-messaging message-bus, as it still relies on fedmsg
- improve it's development state, by improving the test coverage, adding more static-checks e.t.c.
- migrate FMN to the fedora-messaging message-bus, as it still relies on fedmsg
- improve it's development state, by improving the test coverage, adding more
static-checks e.t.c.
Preliminary notes on using other off-the-shelf applications
-----------------------------------------------------------
As we want people to be able to get notified on messages on the bus,
we'd need to integrate with our fedora-messaging infrastructure and our
account system. We didn't find application that could be deployed and
fullfil all of the needs FMN currently serves
As we want people to be able to get notified on messages on the bus, we'd need to
integrate with our fedora-messaging infrastructure and our account system. We didn't
find application that could be deployed and fullfil all of the needs FMN currently
serves
Create a new bespoke application that better suits our needs
------------------------------------------------------------
This is an option we'd take if and only if the path of maintaining the original application
was not feasible, or there was some way to greatly simplify the new implementation.
This is an option we'd take if and only if the path of maintaining the original
application was not feasible, or there was some way to greatly simplify the new
implementation.
With FMN neither is the case.

298
docs/github2fedmsg/enhancing_the_development_environment.rst
View file

@ -1,170 +1,166 @@
.. _enhancing_the_development_environment:
Enhancing the Development Environment
========================================
=====================================
The current state of the development environment of the github2fedmsg project
leaves a lot to be desired. This document attempts to address all the existing
tech debts bound to the github2fedmsg project repository.
The current state of the development environment of the github2fedmsg project leaves a
lot to be desired. This document attempts to address all the existing tech debts bound
to the github2fedmsg project repository.
Investigations
----
--------------
**Continuous Integration**
1. Apparently, there has been a continuous integration configured for the
repository, which was last active on Jun 17, 2020 on
`PR #27 <https://github.com/fedora-infra/github2fedmsg/pull/27>`_.
This continuous integration was set up on a Jenkins instance of Fedora
Infrastructure on the CentOS CI namespace (as deduced by the `details
URL <https://jenkins-fedora-apps.apps.ci.centos.org/job/github2fedmsg/job/PR-27/1/display/redirect>`_)
but is either no longer available or can only be accessed via the Red Hat
VPN. If the former is the case, we would want to reconfigure the
continuous integration in the project repository but if latter is the
case, we would want to ensure that the continuous integration is
accessible to the wider community to promote contributor participation in
the project.
2. In order to set up a continuous integration in the project repository,
`GitHub Actions <https://github.com/features/actions>`_ can be put into
use as the project repository itself is hosted on GitHub, thus allowing
for integration and support to a greater extent. It is also possible to
use Fedora Linux containers in GitHub Actions, as evidenced by their use
in `Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/.github/workflows/main.yml#L10>`_,
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/.github/workflows/main.yaml#L8>`_,
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/.github/workflows/main.yaml#L8>`_,
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/.github/workflows/main.yml#L9>`_
and many more such projects, which would help replicate the production
environment accurately for the relevant build and test jobs. Other
continuous integration options like `Zuul CI <https://zuul-ci.org/>`_,
`Travis CI <https://www.travis-ci.com/>`_ etc. can also be considered on
the basis of particular needs.
1. Apparently, there has been a continuous integration configured for the
repository, which was last active on Jun 17, 2020 on `PR #27
<https://github.com/fedora-infra/github2fedmsg/pull/27>`_. This continuous
integration was set up on a Jenkins instance of Fedora Infrastructure on the
CentOS CI namespace (as deduced by the `details URL
<https://jenkins-fedora-apps.apps.ci.centos.org/job/github2fedmsg/job/PR-27/1/display/redirect>`_)
but is either no longer available or can only be accessed via the Red Hat VPN. If
the former is the case, we would want to reconfigure the continuous integration
in the project repository but if latter is the case, we would want to ensure that
the continuous integration is accessible to the wider community to promote
contributor participation in the project.
2. In order to set up a continuous integration in the project repository, `GitHub
Actions <https://github.com/features/actions>`_ can be put into use as the
project repository itself is hosted on GitHub, thus allowing for integration and
support to a greater extent. It is also possible to use Fedora Linux containers
in GitHub Actions, as evidenced by their use in `Meetbot Logs
<https://github.com/fedora-infra/mote/blob/main/.github/workflows/main.yml#L10>`__,
`CentOS Duffy
<https://github.com/CentOS/duffy/blob/dev/.github/workflows/main.yaml#L8>`_,
`Fedora Messaging Notifier
<https://github.com/fedora-infra/fmn/blob/fmn-next/.github/workflows/main.yaml#L8>`__,
`MDAPI
<https://github.com/fedora-infra/mdapi/blob/develop/.github/workflows/main.yml#L9>`_
and many more such projects, which would help replicate the production
environment accurately for the relevant build and test jobs. Other continuous
integration options like `Zuul CI <https://zuul-ci.org/>`_, `Travis CI
<https://www.travis-ci.com/>`_ etc. can also be considered on the basis of
particular needs.
**Development Environment**
1. A small amount of information about setting up a virtual environment and
installing the project using
`setuptools <https://setuptools.pypa.io/en/latest/>`_
is currently available on the
`README.rst document <https://github.com/fedora-infra/github2fedmsg/blob/develop/README.rst>`_
in the project repository. Although, the current setup works fine in a
specific environment having (now EOLed) Python 2.7.x, it can be useful to
abstract the use of setuptools, using dependency management tools like
`poetry <https://python-poetry.org/>`_,
`pipenv <https://pipenv.pypa.io/en/latest/>`_ etc.
to steer away from the unnecessary attention in creating and updating
the setuptools configuration, facilitating for automatic dependency
updates and maintaining a list of dependencies specific to development,
testing and production scenarios.
2. Maintaining codebase to uphold established quality standards will be of
great help, both, for the convenience of existing contributors and to
help onboard potential ones. In the current setup,
`flake8 <https://flake8.pycqa.org/en/latest/>`_ is
`used <https://github.com/fedora-infra/github2fedmsg/blob/develop/tox.ini#L13>`_
for linting the codebase. In addition to flake8,
`isort <https://isort.readthedocs.io/en/latest/>`_ and
`black <https://black.readthedocs.io/en/stable/>`_ can
be used for sorting of dependency imports and enforcing code styles
wherever necessary. This has been of great help in maintaining code
quality, as evidenced by their uses in
`Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/tox.ini#L27-L56>`_,
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/tox.ini#L25-L41>`_,
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L16-L33>`_,
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L26-L54>`_
and many more such projects.
1. A small amount of information about setting up a virtual environment and
installing the project using `setuptools
<https://setuptools.pypa.io/en/latest/>`_ is currently available on the
`README.rst document
<https://github.com/fedora-infra/github2fedmsg/blob/develop/README.rst>`_ in the
project repository. Although, the current setup works fine in a specific
environment having (now EOLed) Python 2.7.x, it can be useful to abstract the use
of setuptools, using dependency management tools like `poetry
<https://python-poetry.org/>`_, `pipenv <https://pipenv.pypa.io/en/latest/>`_
etc. to steer away from the unnecessary attention in creating and updating the
setuptools configuration, facilitating for automatic dependency updates and
maintaining a list of dependencies specific to development, testing and
production scenarios.
2. Maintaining codebase to uphold established quality standards will be of great
help, both, for the convenience of existing contributors and to help onboard
potential ones. In the current setup, `flake8
<https://flake8.pycqa.org/en/latest/>`__ is `used
<https://github.com/fedora-infra/github2fedmsg/blob/develop/tox.ini#L13>`__ for
linting the codebase. In addition to flake8, `isort
<https://isort.readthedocs.io/en/latest/>`__ and `black
<https://black.readthedocs.io/en/stable/>`__ can be used for sorting of
dependency imports and enforcing code styles wherever necessary. This has been of
great help in maintaining code quality, as evidenced by their uses in `Meetbot
Logs <https://github.com/fedora-infra/mote/blob/main/tox.ini#L27-L56>`__, `CentOS
Duffy <https://github.com/CentOS/duffy/blob/dev/tox.ini#L25-L41>`__, `Fedora
Messaging Notifier
<https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L16-L33>`__, `MDAPI
<https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L26-L54>`__ and many
more such projects.
**Automatic Dependency Updates**
1. Projects such as
`Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/.github/dependabot.yml>`_,
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/.github/dependabot.yml>`_ and
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/.github/dependabot.yml>`_
use `GitHub Dependabot <https://github.com/dependabot>`_ to help
automatically update the dependency lockfile as and when new updates and
vulnerability fixes come out for the unexcluded dependencies, matching
the version requirement mentioned in the project configuration file.
While the former case requires having a valid GitHub Dependabot
configuration in the repository, the latter does not and will work
regardless of having it or not. Being GitHub's own security worker bot,
it has a greater extent of integration with the project repositories
available on GitHub.
2. Projects such as
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/renovate.json>`_
and `Anitya <https://github.com/fedora-infra/anitya/blob/master/renovate.json>`_
use `Renovate <https://github.com/apps/renovate>`_ to help automatically
update the dependency lockfile as and when new updates or vulnerability
fixes come out for the unexcluded dependencies, matching the version
requirement mentioned in the project configuration file. Renovate is
highly flexible with its configuration, allows for checking of dependency
lockfile for verifying checksums, organizing custom schedules for updates
and also supports auto-merging of the dependency update pull requests.
1. Projects such as `Meetbot Logs
<https://github.com/fedora-infra/mote/blob/main/.github/dependabot.yml>`__,
`CentOS Duffy
<https://github.com/CentOS/duffy/blob/dev/.github/dependabot.yml>`__ and `MDAPI
<https://github.com/fedora-infra/mdapi/blob/develop/.github/dependabot.yml>`__
use `GitHub Dependabot <https://github.com/dependabot>`__ to help automatically
update the dependency lockfile as and when new updates and vulnerability fixes
come out for the unexcluded dependencies, matching the version requirement
mentioned in the project configuration file. While the former case requires
having a valid GitHub Dependabot configuration in the repository, the latter does
not and will work regardless of having it or not. Being GitHub's own security
worker bot, it has a greater extent of integration with the project repositories
available on GitHub.
2. Projects such as `Fedora Messaging Notifier
<https://github.com/fedora-infra/fmn/blob/fmn-next/renovate.json>`__ and `Anitya
<https://github.com/fedora-infra/anitya/blob/master/renovate.json>`_ use
`Renovate <https://github.com/apps/renovate>`_ to help automatically update the
dependency lockfile as and when new updates or vulnerability fixes come out for
the unexcluded dependencies, matching the version requirement mentioned in the
project configuration file. Renovate is highly flexible with its configuration,
allows for checking of dependency lockfile for verifying checksums, organizing
custom schedules for updates and also supports auto-merging of the dependency
update pull requests.
**Outdated dependencies**
1. The project makes use of a certain set of dependencies which are either
left in a pre-release state or are no longer actively maintained. While
these dependencies can be put to use in the project in its current state,
it is very likely that a lot of security vulnerabilities must have flown
under the radar unpatched, in the absence of features updates, security
patches and bug fixes. Apart from that, being older libraries/frameworks -
the availability of updated documentation is likely to be scarce for
these, thus making the development using these dependencies more
challenging for the maintainers.
2. Moving forward, we would want to look into either possible replacing the
following dependencies or avoiding their use entirely.
1. The project makes use of a certain set of dependencies which are either left in a
pre-release state or are no longer actively maintained. While these dependencies
can be put to use in the project in its current state, it is very likely that a
lot of security vulnerabilities must have flown under the radar unpatched, in the
absence of features updates, security patches and bug fixes. Apart from that,
being older libraries/frameworks - the availability of updated documentation is
likely to be scarce for these, thus making the development using these
dependencies more challenging for the maintainers.
2. Moving forward, we would want to look into either possible replacing the
following dependencies or avoiding their use entirely.
1. `Velruse <https://pypi.org/project/velruse/>`_, had its last
update almost *9 years back* in
`Aug 30, 2013 <https://pypi.org/project/velruse/1.1.1/>`_.
2. `WebError <https://pypi.org/project/WebError/>`_, had its last update
almost *6 years back* in
`Apr 10, 2016 <https://pypi.org/project/WebError/0.13.1/>`_ and the
`project documentation states <https://github.com/Pylons/weberror/blob/e395f6d0f0becb299b905197c4577d3b74ad40ae/README.rst#notice>`_
that the project is no longer actively maintained.
3. `pyramid-mako <https://pypi.org/project/pyramid-mako/>`_ had its last
update almost *3 years back* in
`Aug 19, 2019 <https://pypi.org/project/pyramid-mako/1.1.0/>`_.
4. `transaction <https://pypi.org/project/transaction/>`_ had its last
update almost *2 years back* in
`Dec 11, 2020 <https://pypi.org/project/transaction/3.0.1/>`_.
1. `Velruse <https://pypi.org/project/velruse/>`_, had its last update almost *9
years back* in `Aug 30, 2013 <https://pypi.org/project/velruse/1.1.1/>`_.
2. `WebError <https://pypi.org/project/WebError/>`_, had its last update almost
*6 years back* in `Apr 10, 2016 <https://pypi.org/project/WebError/0.13.1/>`_
and the `project documentation states
<https://github.com/Pylons/weberror/blob/e395f6d0f0becb299b905197c4577d3b74ad40ae/README.rst#notice>`_
that the project is no longer actively maintained.
3. `pyramid-mako <https://pypi.org/project/pyramid-mako/>`_ had its last update
almost *3 years back* in `Aug 19, 2019
<https://pypi.org/project/pyramid-mako/1.1.0/>`_.
4. `transaction <https://pypi.org/project/transaction/>`_ had its last update
almost *2 years back* in `Dec 11, 2020
<https://pypi.org/project/transaction/3.0.1/>`_.
**Contributor's Quality of Life**
1. Certain additional documentations such as CODE OF CONDUCT, CONTRIBUTING
GUIDELINES etc. can either be added or linked in, to set the expectations
right for both existing and potential contributors. The wiki feature
present in the project repository can be preferred to be put into use
for keeping documentation like these which are less likely to change
across multiple releases. These documentation are available in
repositories such as MDAPI
`[1] <https://github.com/fedora-infra/mdapi/blob/develop/CODE_OF_CONDUCT.md>`_
`[2] <https://github.com/fedora-infra/mdapi/blob/develop/CONTRIBUTING.md>`_,
Meetbot Logs
`[1] <https://github.com/fedora-infra/mote/blob/main/CODE_OF_CONDUCT.md>`_
`[2] <https://github.com/fedora-infra/mote/blob/main/CONTRIBUTING.md>`_,
CentOS Duffy
`[1] <https://github.com/CentOS/duffy/blob/dev/CONTRIBUTING.md>`_ etc.
and can be included here too.
2. Tests and code coverage should be added to ensure that the project code
remains functional over time across the addition of features, fixing of
bugs, patching of security vulnerabilities etc. The
`tests <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/tests.py>`_
currently available in the project repository have a limited scope and
are not automated using continuous integration to test out the code
`on pull requests <https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#pull_request>`_,
`on pushes <https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#push>`_
and
`before deployment workflows <https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#workflow_dispatch>`_.
These tests and code coverage have been implemented in
`Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/tox.ini#L7-L19>`_,
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/tox.ini#L7-L18>`_,
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L7-L14>`_,
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L7-L17>`_
and many more such repositories and are automated using GitHub Actions.
1. Certain additional documentations such as CODE OF CONDUCT, CONTRIBUTING
GUIDELINES etc. can either be added or linked in, to set the expectations right
for both existing and potential contributors. The wiki feature present in the
project repository can be preferred to be put into use for keeping documentation
like these which are less likely to change across multiple releases. These
documentation are available in repositories such as MDAPI `[1]
<https://github.com/fedora-infra/mdapi/blob/develop/CODE_OF_CONDUCT.md>`__ `[2]
<https://github.com/fedora-infra/mdapi/blob/develop/CONTRIBUTING.md>`__, Meetbot
Logs `[1] <https://github.com/fedora-infra/mote/blob/main/CODE_OF_CONDUCT.md>`__
`[2] <https://github.com/fedora-infra/mote/blob/main/CONTRIBUTING.md>`__, CentOS
Duffy `[1] <https://github.com/CentOS/duffy/blob/dev/CONTRIBUTING.md>`__ etc. and
can be included here too.
2. Tests and code coverage should be added to ensure that the project code remains
functional over time across the addition of features, fixing of bugs, patching of
security vulnerabilities etc. The `tests
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/tests.py>`_
currently available in the project repository have a limited scope and are not
automated using continuous integration to test out the code `on pull requests
<https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#pull_request>`_,
`on pushes
<https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#push>`_
and `before deployment workflows
<https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#workflow_dispatch>`_.
These tests and code coverage have been implemented in `Meetbot Logs
<https://github.com/fedora-infra/mote/blob/main/tox.ini#L7-L19>`__, `CentOS Duffy
<https://github.com/CentOS/duffy/blob/dev/tox.ini#L7-L18>`__, `Fedora Messaging
Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L7-L14>`__,
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L7-L17>`__ and
many more such repositories and are automated using GitHub Actions.
**Automating Deployments to OpenShift**
1. The project currently makes use of semantic versioning for its releases
and requires manual intervention for deploying every distinct release to
both `staging <https://apps.fedoraproject.org/github2fedmsg>`_ and
`production <https://apps.stg.fedoraproject.org/github2fedmsg>`_
environments. The effort can be reduced by the use of webhooks which
would help monitor distinct release on the project repository and then
automatically run deployment scripts to put the new version up on
the staging environment. For this, GitHub webhooks can be configured to
be used on the said OpenShift cluster and the production deployment is
suggested to require manual intervention to make way for additional due
diligence before deployment.
1. The project currently makes use of semantic versioning for its releases and
requires manual intervention for deploying every distinct release to both
`staging <https://apps.fedoraproject.org/github2fedmsg>`_ and `production
<https://apps.stg.fedoraproject.org/github2fedmsg>`_ environments. The effort can
be reduced by the use of webhooks which would help monitor distinct release on
the project repository and then automatically run deployment scripts to put the
new version up on the staging environment. For this, GitHub webhooks can be
configured to be used on the said OpenShift cluster and the production deployment
is suggested to require manual intervention to make way for additional due
diligence before deployment.

54
docs/github2fedmsg/index.rst
View file

@ -5,10 +5,10 @@ Purpose
-------
This investigation's goal is to address tech debt of github2fedmsg application.
github2fedmsg application is used to translate events emitted by github to fedora messaging
messages using github webhooks. It also has database of projects to watch and provides web
interface to add new repositories. It authenticates with FAS and Github, linking accounts
together.
github2fedmsg application is used to translate events emitted by github to fedora
messaging messages using github webhooks. It also has database of projects to watch and
provides web interface to add new repositories. It authenticates with FAS and Github,
linking accounts together.
The app has git-hub2fedmsg bot on GitHub that is being used to link the FAS account to
Github.
@ -16,34 +16,35 @@ Github.
Resources
---------
* github2fedmsg initiative proposal: https://pagure.io/cpe/initiatives-proposal/issue/24
* github2fedmsg repo: https://github.com/fedora-infra/github2fedmsg
* github2fedmsg web: https://apps.fedoraproject.org/github2fedmsg
* github2fedmsg ansible: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/github2fedmsg
* github2fedmsg SOP: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/github2fedmsg/
- github2fedmsg initiative proposal: https://pagure.io/cpe/initiatives-proposal/issue/24
- github2fedmsg repo: https://github.com/fedora-infra/github2fedmsg
- github2fedmsg web: https://apps.fedoraproject.org/github2fedmsg
- github2fedmsg ansible:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/github2fedmsg
- github2fedmsg SOP:
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/github2fedmsg/
Requirements
------------
* Porting to Flask and authlib
* Linking GitHub account to FAS account
* Listening to GitHub events
* Translating GitHub events to Fedora messages
* Add/remove repositories to watch
* CI
* Documentation
* OpenShift deployment
- Porting to Flask and authlib
- Linking GitHub account to FAS account
- Listening to GitHub events
- Translating GitHub events to Fedora messages
- Add/remove repositories to watch
- CI
- Documentation
- OpenShift deployment
What Needs To Be Addressed?
---------------------------
* Create message schema
* python2 -> python3
* migration to OpenShift
* Documentation - only README and SOPs are available
* Update development environment
* There is no staging environment
- Create message schema
- python2 -> python3
- migration to OpenShift
- Documentation - only README and SOPs are available
- Update development environment
- There is no staging environment
Investigation
-------------
@ -57,12 +58,11 @@ Here is the list of ideas/things we discussed/looked at:
rewrite
enhancing_the_development_environment
Conclusions
-----------
Investigation team recommends to not work on this app, but instead work
on :doc:`../webhook2fedmsg/index`.
Investigation team recommends to not work on this app, but instead work on
:doc:`../webhook2fedmsg/index`.
Proposed Roadmap
----------------

76
docs/github2fedmsg/revitalization.rst
View file

@ -3,57 +3,71 @@
Revitalization of github2fedmsg application
===========================================
This document will investigate the possibility of updating the code base we currently have.
This document will investigate the possibility of updating the code base we currently
have.
What needs to be addressed (from our index)
-------------------------------------------
**Python 2 -> Python 3**
| *Current situation*: Written in Python 2.7.x which is now `EOLed <https://www.python.org/doc/sunset-python-2/>`_.
| *Suggested outcome*: Python 3.10.x (and above).
| *Reason(s)*: Benefits from performance improvements, bug fixes, security patches etc. of the language.
| *Current situation*: Written in Python 2.7.x which is now `EOLed
<https://www.python.org/doc/sunset-python-2/>`_.
| *Suggested outcome*: Python 3.10.x (and above).
| *Reason(s)*: Benefits from performance improvements, bug fixes, security patches
etc. of the language.
**Migration to OpenShift**
| *Current situation*: The app is running in a Virtual Machine (VM).
| *Suggested outcome*: Deploy it as an OpenShift service.
| *Reason(s)*: Automated deployments and we already have an OpenShift cluster.
| *Current situation*: The app is running in a Virtual Machine (VM).
| *Suggested outcome*: Deploy it as an OpenShift service.
| *Reason(s)*: Automated deployments and we already have an OpenShift cluster.
**Documentation**
| *Current situation*: Only a brief README file and `SOP <https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/github2fedmsg/>`_ are available.
| *Suggested outcome*: It would be great if the documentation was more extensive. It should include instructions about setting up the development environment, contributing guidelines, code of conduct, etc. This is also covered in the `dev environment enhancement doc <https://fedora-arc.readthedocs.io/en/latest/github2fedmsg/enhancing_the_development_environment.html>`_.
| *Reason(s)*: Easier onboarding and setting expectations for contributors.
| *Current situation*: Only a brief README file and `SOP
<https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/github2fedmsg/>`_ are
available.
| *Suggested outcome*: It would be great if the documentation was more extensive.
It should include instructions about setting up the development environment,
contributing guidelines, code of conduct, etc. This is also covered in the `dev
environment enhancement doc
<https://fedora-arc.readthedocs.io/en/latest/github2fedmsg/enhancing_the_development_environment.html>`_.
| *Reason(s)*: Easier onboarding and setting expectations for contributors.
**Porting to Authlib**
| *Current outcome*: The authentication library used is `Velruse <https://velruse.readthedocs.io/en/latest/usage.html>`_.
| *Suggested using*: `Authlib <https://authlib.org/>`_.
| *Reason(s)*: Velruse hasn't been updated in 9 years and Authlib is in active development.
| *Current outcome*: The authentication library used is `Velruse
<https://velruse.readthedocs.io/en/latest/usage.html>`_.
| *Suggested using*: `Authlib <https://authlib.org/>`_.
| *Reason(s)*: Velruse hasn't been updated in 9 years and Authlib is in active
development.
**Development environment**
| *Current situation*: This is covered in depth in our other document about enhancing the development environment.
| *Current situation*: This is covered in depth in our other document about
enhancing the development environment.
Notes
-----
* setuptools>=58 `breaks support for 2to3 <https://setuptools.pypa.io/en/latest/history.html#v58-0-0>`_.
* 2to3 is `deprecated since Python 3.11 and will be removed in Python 3.13 <https://docs.python.org/3/library/2to3.html>`_.
- setuptools>=58 `breaks support for 2to3
<https://setuptools.pypa.io/en/latest/history.html#v58-0-0>`_.
- 2to3 is `deprecated since Python 3.11 and will be removed in Python 3.13
<https://docs.python.org/3/library/2to3.html>`_.
The Good and the Bad
--------------------
**Advantages**
1. The project is written in (now EOLed) Python 2.7.x
and as Python 3 is currently in active development, the project can
benefit from the performance improvements, bug fixes, and security patches
included in the language itself.
2. The project makes use of a certain set of dependencies, the support of
which has not been ported from Python 2 to Python 3. Porting into
Python 3 would necessitate the use of newer dependencies which would be
more updated, secure and maintained.
1. The project is written in (now EOLed) Python 2.7.x and as Python 3 is currently
in active development, the project can benefit from the performance improvements,
bug fixes, and security patches included in the language itself.
2. The project makes use of a certain set of dependencies, the support of which has
not been ported from Python 2 to Python 3. Porting into Python 3 would
necessitate the use of newer dependencies which would be more updated, secure and
maintained.
**Disadvantages**
1. The move from Python 2 to Python 3 would require special attention into
finding the alternatives for the dependencies which either support only
Python 2 or are unmaintained as of 2022. Inability to find those would
render the rewrite unsuccessful.
2. We will continue using the `Pyramid web framework <https://trypyramid.com/>`_,
however the team lacks the expertise with it. This makes the app difficult
to maintain and team members would have to upskill in this area.
1. The move from Python 2 to Python 3 would require special attention into finding
the alternatives for the dependencies which either support only Python 2 or are
unmaintained as of 2022. Inability to find those would render the rewrite
unsuccessful.
2. We will continue using the `Pyramid web framework <https://trypyramid.com/>`_,
however the team lacks the expertise with it. This makes the app difficult to
maintain and team members would have to upskill in this area.

180
docs/github2fedmsg/rewrite.rst
View file

@ -1,111 +1,135 @@
.. _rewrite:
Complete Rewrite of github2fedmsg Application
====
=============================================
This document will investigate the possibility of rewriting github2fedmsg from scratch.
Elements that could use our attention
----
-------------------------------------
**Project configuration**
| *Currently uses*: Setuptools (`setup.py <https://github.com/fedora-infra/github2fedmsg/blob/develop/setup.py>`_)
| *Suggested using*: Poetry (`pyproject.toml <https://python-poetry.org/docs/pyproject/>`_)
| *Reason(s)*: Easier dependency management, project configuration, testing information etc.
| *Currently uses*: Setuptools (`setup.py
<https://github.com/fedora-infra/github2fedmsg/blob/develop/setup.py>`_)
| *Suggested using*: Poetry (`pyproject.toml
<https://python-poetry.org/docs/pyproject/>`_)
| *Reason(s)*: Easier dependency management, project configuration, testing
information etc.
**Backend language**
| *Currently uses*: Python 2.7.x (now `EOLed <https://www.python.org/doc/sunset-python-2/>`_)
| *Suggested using*: Python 3.10.x (and above)
| *Reason(s)*: Benefit from performance improvements, bug fixes, security patches etc. on the language
| *Currently uses*: Python 2.7.x (now `EOLed
<https://www.python.org/doc/sunset-python-2/>`_)
| *Suggested using*: Python 3.10.x (and above)
| *Reason(s)*: Benefit from performance improvements, bug fixes, security patches
etc. on the language
**Base framework**
| *Currently uses*: `Pyramid <https://trypyramid.com/>`_
| *Suggested using*: `Flask <https://flask.palletsprojects.com/en/2.2.x/>`_ or `FastAPI <https://fastapi.tiangolo.com/>`_
| *Reason(s)*: More knowledge about the suggested ones in the team, streamlined development, more features etc.
| *Currently uses*: `Pyramid <https://trypyramid.com/>`_
| *Suggested using*: `Flask <https://flask.palletsprojects.com/en/2.2.x/>`_ or
`FastAPI <https://fastapi.tiangolo.com/>`_
| *Reason(s)*: More knowledge about the suggested ones in the team, streamlined
development, more features etc.
**Templating implementation**
| *Currently uses*: `Mako <https://www.makotemplates.org/>`_
| *Suggested using*: `Jinja <https://jinja.palletsprojects.com/en/3.1.x/>`_
| *Reason(s)*: More knowledge about the suggested ones in the team, standardized templating etc.
| *Currently uses*: `Mako <https://www.makotemplates.org/>`_
| *Suggested using*: `Jinja <https://jinja.palletsprojects.com/en/3.1.x/>`_
| *Reason(s)*: More knowledge about the suggested ones in the team, standardized
templating etc.
**Authentication library**
| *Currently uses*: `Velruse <https://velruse.readthedocs.io/en/latest/usage.html>`_
| *Suggested using*: `Authlib <https://authlib.org/>`_
| *Reason(s)*: Authlib is actively maintained while Velruse had its last update `9 years ago <https://pypi.org/project/velruse/1.1.1/>`_.
| *Currently uses*: `Velruse
<https://velruse.readthedocs.io/en/latest/usage.html>`_
| *Suggested using*: `Authlib <https://authlib.org/>`_
| *Reason(s)*: Authlib is actively maintained while Velruse had its last update `9
years ago <https://pypi.org/project/velruse/1.1.1/>`_.
**Frontend CSS library**
| *Currently uses*: `Bootstrap 3.1.1 (Fedora) <https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/bootstrap-3.1.1-fedora>`_
| *Suggested using*: Updated version of Bootstrap for eg. `Boostrap 5 <https://getbootstrap.com/docs/5.0/getting-started/introduction/>`_
| *Reason(s)*: Web interface looks/feels dated and is not responsive to variable widths.
| *Currently uses*: `Bootstrap 3.1.1 (Fedora)
<https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/bootstrap-3.1.1-fedora>`_
| *Suggested using*: Updated version of Bootstrap for eg. `Boostrap 5
<https://getbootstrap.com/docs/5.0/getting-started/introduction/>`_
| *Reason(s)*: Web interface looks/feels dated and is not responsive to variable
widths.
**Notification popups**
| *Currently uses*: `Messenger 1.4.1 <https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/messenger-1.4.1>`_
| *Suggested using*: Updated version of Messenger or any maintained alternative
| *Reason(s)*: Library was last added `8 years back <https://github.com/fedora-infra/github2fedmsg/commit/cb98f6c7b74bb50c7cb8ee58aff81033c23300ee>`_ and could use an update.
| *Currently uses*: `Messenger 1.4.1
<https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/messenger-1.4.1>`_
| *Suggested using*: Updated version of Messenger or any maintained alternative
| *Reason(s)*: Library was last added `8 years back
<https://github.com/fedora-infra/github2fedmsg/commit/cb98f6c7b74bb50c7cb8ee58aff81033c23300ee>`_
and could use an update.
**Interface interactivity**
| *Currently uses*: `JQuery 1.11.0 <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/static/jquery-1.11.0.min.js>`_
| *Suggested using*: Updated version of JQuery for eg. `JQuery 3.6.1 <https://code.jquery.com/jquery-3.6.1.min.js>`_
| *Reason(s)*: Library was last added `9 years back <https://github.com/fedora-infra/github2fedmsg/commit/8985732f1e22a565dfd3ce9964896e9e4f86657e>`_ and could use an update.
| *Currently uses*: `JQuery 1.11.0
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/static/jquery-1.11.0.min.js>`_
| *Suggested using*: Updated version of JQuery for eg. `JQuery 3.6.1
<https://code.jquery.com/jquery-3.6.1.min.js>`_
| *Reason(s)*: Library was last added `9 years back
<https://github.com/fedora-infra/github2fedmsg/commit/8985732f1e22a565dfd3ce9964896e9e4f86657e>`_
and could use an update.
**API modelling**
| *Currently uses*: In-place `creation of data structures <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/views/webhooks.py#L247-L253>`_ in worker functions
| *Suggested using*: Data validation and typing libraries like `Pydantic <https://pydantic-docs.helpmanual.io/>`_
| *Reason(s)*: Better codebase modularity and readability, auto-validation of API data types etc.
| *Currently uses*: In-place `creation of data structures
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/views/webhooks.py#L247-L253>`_
in worker functions
| *Suggested using*: Data validation and typing libraries like `Pydantic
<https://pydantic-docs.helpmanual.io/>`_
| *Reason(s)*: Better codebase modularity and readability, auto-validation of API
data types etc.
**Command line parsing**
| *Currently uses*: None. Just making do with `stdio` and `stdout`.
| *Suggested using*: `Click <https://click.palletsprojects.com/en/8.1.x/>`_ or any other alternatives
| *Reason(s)*: Easy-to-use command line creation, handling of options and commands with auto-generated help etc.
| *Currently uses*: None. Just making do with `stdio` and `stdout`.
| *Suggested using*: `Click <https://click.palletsprojects.com/en/8.1.x/>`_ or any
other alternatives
| *Reason(s)*: Easy-to-use command line creation, handling of options and commands
with auto-generated help etc.
The good and the bad
----
--------------------
**Advantages**
1. In its current state, the project is written in (now EOLed) Python 2.7.x
and as Python 3 is currently in active development, the project can
benefit from the performance improvements, bug fixes and security patches
included in the language itself.
2. The project makes use of a certain set of dependencies, the support of
which, have not been ported from Python 2 to Python 3. Porting into
Python 3 would necessitate the use of newer dependencies which would be
more updated, secure and maintained.
3. In its current state, the project makes use of the
`Pyramid <https://trypyramid.com/>`_ web framework, the expertise of which
the team currently lacks, making it unmaintainable right now. A rewrite
using libraries and frameworks that the team is acquainted with would help.
4. The web interface makes use of non-default
`Mako <https://www.makotemplates.org/>`_-based
`templates <https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/templates>`_
that work right now but due to their obscure nature, it can become
difficult to debug the templates later. A rewrite of templates using
a standardized
`Jinja <https://jinja.palletsprojects.com/en/3.1.x/>`_ format would go a
long way.
5. The web interface could use updated versions (or replacements) of
unmaintained JavaScript and CSS3 dependencies (for now, it has
`Bootstrap 3.1.1 <https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/bootstrap-3.1.1-fedora>`_,
`Messenger 1.4.1 <https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/messenger-1.4.1>`_,
`JQuery 1.11.0 <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/static/jquery-1.11.0.min.js>`_
etc.) in order to introduce responsiveness and provide a better look and
feel.
6. The API model can be standardized with the use of typing and data
validation libraries such as
`Pydantic <https://pydantic-docs.helpmanual.io/>`_ instead of
`forming data structures <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/views/webhooks.py#L247-L253>`_
in the worker function itself, making the code more modular and
maintainable in the long run.
1. In its current state, the project is written in (now EOLed) Python 2.7.x and as
Python 3 is currently in active development, the project can benefit from the
performance improvements, bug fixes and security patches included in the language
itself.
2. The project makes use of a certain set of dependencies, the support of which,
have not been ported from Python 2 to Python 3. Porting into Python 3 would
necessitate the use of newer dependencies which would be more updated, secure and
maintained.
3. In its current state, the project makes use of the `Pyramid
<https://trypyramid.com/>`_ web framework, the expertise of which the team
currently lacks, making it unmaintainable right now. A rewrite using libraries
and frameworks that the team is acquainted with would help.
4. The web interface makes use of non-default `Mako
<https://www.makotemplates.org/>`_-based `templates
<https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/templates>`_
that work right now but due to their obscure nature, it can become difficult to
debug the templates later. A rewrite of templates using a standardized `Jinja
<https://jinja.palletsprojects.com/en/3.1.x/>`_ format would go a long way.
5. The web interface could use updated versions (or replacements) of unmaintained
JavaScript and CSS3 dependencies (for now, it has `Bootstrap 3.1.1
<https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/bootstrap-3.1.1-fedora>`_,
`Messenger 1.4.1
<https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/messenger-1.4.1>`_,
`JQuery 1.11.0
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/static/jquery-1.11.0.min.js>`_
etc.) in order to introduce responsiveness and provide a better look and feel.
6. The API model can be standardized with the use of typing and data validation
libraries such as `Pydantic <https://pydantic-docs.helpmanual.io/>`_ instead of
`forming data structures
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/views/webhooks.py#L247-L253>`_
in the worker function itself, making the code more modular and maintainable in
the long run.
**Disadvantages**
1. The move from Python 2 to Python 3 would require special attention into
the finding alternatives for the dependencies which either support only
Python 2 or are unmaintained as of 2022. Inability to find those would
render the rewrite unsuccessful.
2. In the wake of new ways to interact with an API, some features (eg. web
interface, when API is suggested to be used by itself) of the current
version have become redundant and a 1:1 rewrite would not help as it
would frontport those features too.
3. While reworking the database model, it is possible that the existing dump
cannot be fully imported due to the changes introduced in the database
schema to better work the newer reimplementation, thereby potentially
causing data loss.
1. The move from Python 2 to Python 3 would require special attention into the
finding alternatives for the dependencies which either support only Python 2 or
are unmaintained as of 2022. Inability to find those would render the rewrite
unsuccessful.
2. In the wake of new ways to interact with an API, some features (eg. web
interface, when API is suggested to be used by itself) of the current version
have become redundant and a 1:1 rewrite would not help as it would frontport
those features too.
3. While reworking the database model, it is possible that the existing dump cannot
be fully imported due to the changes introduced in the database schema to better
work the newer reimplementation, thereby potentially causing data loss.

15
docs/index.rst
View file

@ -2,17 +2,14 @@ Welcome to ARC notes's documentation!
=====================================
.. toctree::
:maxdepth: 2
:caption: Contents:
:maxdepth: 2
:caption: Contents:
The ARC team is the Advance Reconaissance Crew. A small team within
`CPE <https://docs.fedoraproject.org/en-US/cpe/>`_ aiming at investigating
technical solutions to the `initiatives <https://docs.fedoraproject.org/en-US/cpe/initiatives/>`_
The ARC team is the Advance Reconaissance Crew. A small team within `CPE
<https://docs.fedoraproject.org/en-US/cpe/>`_ aiming at investigating technical
solutions to the `initiatives <https://docs.fedoraproject.org/en-US/cpe/initiatives/>`_
submitted.
Contents:
.. toctree::
@ -22,7 +19,5 @@ Contents:
initiatives
arc_names
The sources for this documentation can be found at: https://pagure.io/fedora-infra/arc
in the ``docs`` folder.

3
docs/initiatives.rst
View file

@ -6,6 +6,7 @@ Initiatives
Drafts
------
.. toctree::
:maxdepth: 1
@ -13,6 +14,7 @@ Drafts
Completed review
----------------
.. toctree::
:maxdepth: 1
@ -30,6 +32,7 @@ Completed review
Implemented
-----------
.. toctree::
:maxdepth: 1

78
docs/kerneltest/continue_from_v2.rst
View file

@ -3,70 +3,68 @@
Use v2 Code Of kerneltest Application
=====================================
There is already a work in progress rewrite of kerneltest application started by Jeremy Cline,
which was never finished. This document is investigating if this option is feasible
and we should continue on the code already written.
From meeting with Justin Forbes this code is not 100% ready, which is the reason it was never
been released.
There is already a work in progress rewrite of kerneltest application started by Jeremy
Cline, which was never finished. This document is investigating if this option is
feasible and we should continue on the code already written. From meeting with Justin
Forbes this code is not 100% ready, which is the reason it was never been released.
What is in v2
-------------
* Some unit tests in `kerneltest/tests` (not covering everything,
but at least part of the code is covered) run by tox - failing
with never version of libraries
* API calls for `/api/v1/results`
- Some unit tests in `kerneltest/tests` (not covering everything, but at least part of
the code is covered) run by tox - failing with never version of libraries
- API calls for `/api/v1/results`
* GET method to retrieve tests information
* POST method to add new test run to db
(authentication scopes doesn't seem to be set correctly)
* Sending Fedora message when adding a new test run
- GET method to retrieve tests information
- POST method to add new test run to db (authentication scopes doesn't seem to be set
correctly)
- Sending Fedora message when adding a new test run
* Basic OIDC implementation using `flask_oidc` library
* Configuration file loading and defaults
* Forms for updating/creating release and submitting logs are in place
* Web routes implemented
- Basic OIDC implementation using `flask_oidc` library
- Configuration file loading and defaults
- Forms for updating/creating release and submitting logs are in place
- Web routes implemented
* `/index`
* `/login`
* `/logout`
* `/release/<release>` - information about specific fedora release
* `/kernel/<kernel>` - information about specific kernel version
* `/results/<test_run_id>` - information about specific test
* `/upload` - test result uploading
* `/stats`
* `/admin/new` - add new release (does emit fedora message)
* `/admin/<release>/edit` - edit existing release (does emit fedora message)
- `/index`
- `/login`
- `/logout`
- `/release/<release>` - information about specific fedora release
- `/kernel/<kernel>` - information about specific kernel version
- `/results/<test_run_id>` - information about specific test
- `/upload` - test result uploading
- `/stats`
- `/admin/new` - add new release (does emit fedora message)
- `/admin/<release>/edit` - edit existing release (does emit fedora message)
* DB models for Test, TestRun and Release
- DB models for Test, TestRun and Release
What is missing in v2
---------------------
* Documentation
* No support for uploading logs yet
* Web interface revitalization
* Code itself more than 3 years old, there could be some issue with newer libraries
- Documentation
- No support for uploading logs yet
- Web interface revitalization
- Code itself more than 3 years old, there could be some issue with newer libraries
Notes
-----
* There is a `kerneltest/harness.py` source file in the v2. This could be ignored as it's
maintained separately and isn't part of the kerneltest application.
* The Vagrantfile introduced with v2 rewrite doesn't start with vagrant 2.2.19, see
- There is a `kerneltest/harness.py` source file in the v2. This could be ignored as
it's maintained separately and isn't part of the kerneltest application.
- The Vagrantfile introduced with v2 rewrite doesn't start with vagrant 2.2.19, see
https://github.com/hashicorp/vagrant/issues/12878
* Ansible for vagrant provisioning is missing installation of some packages for database
- Ansible for vagrant provisioning is missing installation of some packages for database
creation (python-alembic, python-toml)
* Ansible for vagrant provisioning doesn't create config file. Because of this creation of
the database fails
* The current state of v2 can't be even started, failing on newer version of wtforms library
- Ansible for vagrant provisioning doesn't create config file. Because of this creation
of the database fails
- The current state of v2 can't be even started, failing on newer version of wtforms
library
The Good Points
---------------
1. Not starting from scratch = less work
The Bad points
--------------

58
docs/kerneltest/current_v1.2.1.rst
View file

@ -3,54 +3,52 @@
Current Live Version of Kernel Test App v1.2.1
==============================================
The current version of the Kernel Test App running is v1.2.1
The code is no longer maintained and stale since 2017. There was an attempted
rewrite but this was never put live
The current version of the Kernel Test App running is v1.2.1 The code is no longer
maintained and stale since 2017. There was an attempted rewrite but this was never put
live
Resources
---------
Release 1.2.1: https://github.com/jmflinuxtx/kerneltest-harness/tree/c2d807506e2a9ac3ca6bc5b60405b5700a72719c
Release 1.2.1:
https://github.com/jmflinuxtx/kerneltest-harness/tree/c2d807506e2a9ac3ca6bc5b60405b5700a72719c
What is in v1.21
-------------
----------------
* Tests are in the `tests` folder and appear to cover most functionality
* Uses flask `fas_openid` library for authentication
* Configuration file loading and defaults
* Forms for updating/creating release and submitting logs are in place
* Web routes implemented
* `/` - index
* `/release/<release>` - info about release version specified
* `/kernel/<kernel>` - info about kernel version specified
* `/logs/<logid>`
* `/stats`
* `/upload`
* `/upload/autotest`
* `/upload/anonymous`
* `/login`
* `/logout`
* `/admin/new` - Add new release
* `/admin/<relnum>/edit` - edit existing release
- Tests are in the `tests` folder and appear to cover most functionality
- Uses flask `fas_openid` library for authentication
- Configuration file loading and defaults
- Forms for updating/creating release and submitting logs are in place
- Web routes implemented
- `/` - index
- `/release/<release>` - info about release version specified
- `/kernel/<kernel>` - info about kernel version specified
- `/logs/<logid>`
- `/stats`
- `/upload`
- `/upload/autotest`
- `/upload/anonymous`
- `/login`
- `/logout`
- `/admin/new` - Add new release
- `/admin/<relnum>/edit` - edit existing release
Notes
-----
* There is no documentation
* This is the live version and it works as is
* Web interface outdated
* Old unmaintained codebase(all functionality in single file)
* Local vagrant solution uses outdated version of Fedora and doesn't currently
work
- There is no documentation
- This is the live version and it works as is
- Web interface outdated
- Old unmaintained codebase(all functionality in single file)
- Local vagrant solution uses outdated version of Fedora and doesn't currently work
The Good Points
---------------
1. The application works as expected and provides necessary functionality
The Bad points
--------------

16
docs/kerneltest/enhancement_of_repository.rst
View file

@ -3,15 +3,15 @@
Enhancement of dev env for kerneltest
=====================================
There is plenty of tech debt regarding the kerneltest repository. This document
will try to address all the current tech debt tied to kerneltest repository.
There is plenty of tech debt regarding the kerneltest repository. This document will try
to address all the current tech debt tied to kerneltest repository.
Investigation
-------------
* CI - Currently using Travis CI, which is no longer working. Recommending to switch
to either GitHub action or use Zuul CI
* Development environment - There is a vagrant using F25, needs to be updated
* Automatic dependency updater - recommending renovate
* When moving to OpenShift we can consolidate adding automatic deployment with
GitHub webhooks
- CI - Currently using Travis CI, which is no longer working. Recommending to switch to
either GitHub action or use Zuul CI
- Development environment - There is a vagrant using F25, needs to be updated
- Automatic dependency updater - recommending renovate
- When moving to OpenShift we can consolidate adding automatic deployment with GitHub
webhooks

79
docs/kerneltest/index.rst
View file

@ -4,48 +4,47 @@ kerneltest
Purpose
-------
This investigation's goal is to address tech debt of kerneltest application.
kerneltest applications is used by Fedora QA for submitting test logs for
kernel builds in Fedora. It also provides web interface for users to access
and upload logs.
Currently there is a partial rewrite v2 in repository, which is not functionality
complete and was never released.
This investigation's goal is to address tech debt of kerneltest application. kerneltest
applications is used by Fedora QA for submitting test logs for kernel builds in Fedora.
It also provides web interface for users to access and upload logs. Currently there is a
partial rewrite v2 in repository, which is not functionality complete and was never
released.
Resources
---------
* kerneltest repo: https://github.com/jmflinuxtx/kerneltest-harness
* kerneltest web: https://apps.fedoraproject.org/kerneltest/
* kerneltest ansible: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/kerneltest
* kerneltest wiki: http://fedoraproject.org/wiki/KernelTestingInitiative
- kerneltest repo: https://github.com/jmflinuxtx/kerneltest-harness
- kerneltest web: https://apps.fedoraproject.org/kerneltest/
- kerneltest ansible:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/kerneltest
- kerneltest wiki: http://fedoraproject.org/wiki/KernelTestingInitiative
Requirements
------------
* Be able to upload logs anonymously, authenticated, super user
* Superuser should use api token or similar(no password login required)
* Be able to view logs on front page showing super user ones
* When clicking on the release it should show all logs for a release
* Stats: https://apps.fedoraproject.org/kerneltest/stats (currently stored in postgres)
* Logs needs to be saved in persistent volume
* Send Fedora Messages for badges
* Size limitation on the uploaded logs (currently hardcoded in code)
* Documentation
* OpenShift deployment
- Be able to upload logs anonymously, authenticated, super user
- Superuser should use api token or similar(no password login required)
- Be able to view logs on front page showing super user ones
- When clicking on the release it should show all logs for a release
- Stats: https://apps.fedoraproject.org/kerneltest/stats (currently stored in postgres)
- Logs needs to be saved in persistent volume
- Send Fedora Messages for badges
- Size limitation on the uploaded logs (currently hardcoded in code)
- Documentation
- OpenShift deployment
What Needs To Be Addressed?
---------------------------
* Update authentication library
* Switch CI
* fedmsg -> fedora messaging (some v2 parts could be used)
* python2 -> python3
* migration to OpenShift
* Documentation - Fedora QA has some
* Users don't receive any badges
* Update development environment
* Web interface (should be revitalized)
- Update authentication library
- Switch CI
- fedmsg -> fedora messaging (some v2 parts could be used)
- python2 -> python3
- migration to OpenShift
- Documentation - Fedora QA has some
- Users don't receive any badges
- Update development environment
- Web interface (should be revitalized)
Investigation
-------------
@ -60,25 +59,23 @@ Here is the list of ideas/things we discussed/looked at:
continue_from_v2
enhancement_of_repository
Conclusions
-----------
The best approach for the kerneltest initiative will be the complete rewrite
with salvaging what could be salvaged from v1.2.1 and v2 rewrite.
The best approach for the kerneltest initiative will be the complete rewrite with
salvaging what could be salvaged from v1.2.1 and v2 rewrite.
Proposed Roadmap
----------------
* Step 1 - Update the development environment
* Step 2 - Rewrite the kerneltest app with tests + documentation
* Step 3 - OpenShift migration
* Step 4 - Database migration
- Step 1 - Update the development environment
- Step 2 - Rewrite the kerneltest app with tests + documentation
- Step 3 - OpenShift migration
- Step 4 - Database migration
Estimate of work
----------------
A complete rewrite of the backend and Openshift deployment is estimated to take
3 developers 12 weeks.
The frontend is estimated to take 3 developers 8 weeks, although it may be a
project the Fedora Websites and Apps team could consider.
A complete rewrite of the backend and Openshift deployment is estimated to take 3
developers 12 weeks. The frontend is estimated to take 3 developers 8 weeks, although it
may be a project the Fedora Websites and Apps team could consider.

12
docs/kerneltest/rewrite_of_kerneltest_app.rst
View file

@ -4,19 +4,18 @@ Complete Rewrite Of kerneltest Application
==========================================
One of the options is to completely rewrite the current solution and start from scratch.
This will allow us to address all the requirements and get rid of all the existing tech debt.
We can also reuse some code from v2 rewrite.
This will allow us to address all the requirements and get rid of all the existing tech
debt. We can also reuse some code from v2 rewrite.
Requirements
------------
See `https://fedora-arc.readthedocs.io/en/latest/kerneltest/index.html#requirements`_.
See https://fedora-arc.readthedocs.io/en/latest/kerneltest/index.html#requirements.
There are also a few other things that need to be addressed:
* OpenShift deployment
* Database migration (in case the DB schema will change)
- OpenShift deployment
- Database migration (in case the DB schema will change)
The Good Points
---------------
@ -25,7 +24,6 @@ The Good Points
2. More freedom in implementation
3. It will be easier to write documentation
The Bad points
--------------

18
docs/mailman3/add_centos_to_fedora_server.rst
View file

@ -3,19 +3,18 @@
Add Centos Mailing List To Fedora Server
========================================
This was offered as a potential shortcut solution to quickly get a web front end
for the CentOS mailing list. Fedora currently runs a mailman 3 server with
Postorius and HyperKitty which provide a web user interface. The idea would be
to add the CentOS mailing lists to the currently running mailing list.
This was offered as a potential shortcut solution to quickly get a web front end for the
CentOS mailing list. Fedora currently runs a mailman 3 server with Postorius and
HyperKitty which provide a web user interface. The idea would be to add the CentOS
mailing lists to the currently running mailing list.
Basic Implementation
====================
A new domain would be set up for CentOS on the Fedora mailman server. The CentOS
A new domain would be set up for CentOS on the Fedora mailman server. The CentOS
Archives and lists would then be imported to the Fedora Server. The Centos MX domain
would then be pointed here. Configuration would need to be edited to support this.
All email registered would need to be converted to users to integrate with sso logins.
would then be pointed here. Configuration would need to be edited to support this. All
email registered would need to be converted to users to integrate with sso logins.
The Good Points
===============
@ -23,10 +22,9 @@ The Good Points
There are some advantages to this solution.
1. It would involve the least work to implement
2. CentOS and Fedora would only have a single mail server rather than one each to
2. CentOS and Fedora would only have a single mail server rather than one each to
maintain
The Bad Points
==============

89
docs/mailman3/building_rpm_centos_stream.rst
View file

@ -6,23 +6,24 @@ Building RPM for CentOS Stream 8
As part of your investigation ARC team tried to built mailman3 from Fedora `src.rpm` in
CentOS Stream 8 chroot.
This proven to be difficult and it's not possible without providing some patches to fix tests
for mailman3.
This proven to be difficult and it's not possible without providing some patches to fix
tests for mailman3.
Resources
---------
* mailman3 COPR repository: https://copr.fedorainfracloud.org/coprs/zlopez/mailman3
- mailman3 COPR repository: https://copr.fedorainfracloud.org/coprs/zlopez/mailman3
Building RPM
------------
We started with
`newest passing build <https://kojipkgs.fedoraproject.org//packages/mailman3/3.3.4/1.fc35/src/mailman3-3.3.4-1.fc35.src.rpm>`_
We started with `newest passing build
<https://kojipkgs.fedoraproject.org//packages/mailman3/3.3.4/1.fc35/src/mailman3-3.3.4-1.fc35.src.rpm>`_
in Fedora koji and tried to build it in CBS using infra tag `infra8-infra-common-el8`
::
cbs build --scratch infra8-infra-common-el8 mailman3-3.3.4-1.fc35.src.rpm
.. code-block::
cbs build --scratch infra8-infra-common-el8 mailman3-3.3.4-1.fc35.src.rpm
This failed with plenty of missing dependencies. Because of this we decided to use COPR
for further building.
@ -30,49 +31,55 @@ for further building.
The `COPR project <https://copr.fedorainfracloud.org/coprs/zlopez/mailman3>`_ used
`centos-stream-8` chroot with EPEL8 repository enabled.
In COPR project we started by building the same `src.rpm` and ended up with list of missing
dependencies::
In COPR project we started by building the same `src.rpm` and ended up with list of
missing dependencies:
No matching package to install: 'python3-aiosmtpd >= 1.4.1'
No matching package to install: 'python3-authheaders >= 0.9.2'
No matching package to install: 'python3-click >= 7.0'
No matching package to install: 'python3-falcon > 1.0.0'
No matching package to install: 'python3-flufl-bounce'
No matching package to install: 'python3-flufl-i18n >= 2.0.1'
No matching package to install: 'python3-lazr-config'
No matching package to install: 'python3-lazr-smtptest'
No matching package to install: 'python3-zope-interface >= 5.0'
.. code-block::
When building the `python-aiosmtpd <https://src.fedoraproject.org/rpms/python-aiosmtpd>`_ project
from `Fedora dist-git <https://src.fedoraproject.org/rpms>`_, we ended up in state, where we would
need to build the whole python stack (python-setuptools, python-pip etc.) to continue.
No matching package to install: 'python3-aiosmtpd >= 1.4.1'
No matching package to install: 'python3-authheaders >= 0.9.2'
No matching package to install: 'python3-click >= 7.0'
No matching package to install: 'python3-falcon > 1.0.0'
No matching package to install: 'python3-flufl-bounce'
No matching package to install: 'python3-flufl-i18n >= 2.0.1'
No matching package to install: 'python3-lazr-config'
No matching package to install: 'python3-lazr-smtptest'
No matching package to install: 'python3-zope-interface >= 5.0'
We decided this wouldn't work for us and started to instead build the
`oldest commit of mailman3 <https://src.fedoraproject.org/rpms/mailman3/c/e7a19ba9fe17d469b590b69441aec83411874fd3?branch=rawhide>`_
from Fedora dist-git. We now need to satisfy lesser amount of dependencies::
When building the `python-aiosmtpd
<https://src.fedoraproject.org/rpms/python-aiosmtpd>`_ project from `Fedora dist-git
<https://src.fedoraproject.org/rpms>`_, we ended up in state, where we would need to
build the whole python stack (python-setuptools, python-pip etc.) to continue.
No matching package to install: 'python3-aiosmtpd'
No matching package to install: 'python3-falcon >= 1.0.0'
No matching package to install: 'python3-flufl-bounce'
No matching package to install: 'python3-flufl-i18n'
No matching package to install: 'python3-lazr-config'
No matching package to install: 'python3-lazr-smtptest'
We decided this wouldn't work for us and started to instead build the `oldest commit of
mailman3
<https://src.fedoraproject.org/rpms/mailman3/c/e7a19ba9fe17d469b590b69441aec83411874fd3?branch=rawhide>`_
from Fedora dist-git. We now need to satisfy lesser amount of dependencies:
.. code-block::
No matching package to install: 'python3-aiosmtpd'
No matching package to install: 'python3-falcon >= 1.0.0'
No matching package to install: 'python3-flufl-bounce'
No matching package to install: 'python3-flufl-i18n'
No matching package to install: 'python3-lazr-config'
No matching package to install: 'python3-lazr-smtptest'
After some trial and error we got all the dependencies satisfied and all of them were
`successfully build in COPR project <https://copr.fedorainfracloud.org/coprs/zlopez/mailman3/packages/>`_.
`successfully build in COPR project
<https://copr.fedorainfracloud.org/coprs/zlopez/mailman3/packages/>`_.
Rebuilding the mailman3 after this failed on tests. After trying newer commits of mailman3
to fix the failing tests and building another missing dependencies, we ended up on the commit
where we started and for which the python stack needs to be rebuild.
Before decision to cease further building attempts we built mailman3 from this
`commit <https://src.fedoraproject.org/rpms/mailman3/c/1e92c38ebbf2dea3af67c9a29f77c1d8b6c31df8?branch=rawhide>`_
in Fedora dist-git. This build has all dependencies satisfied but the
`tests are failing <https://download.copr.fedorainfracloud.org/results/zlopez/mailman3/centos-stream-8-x86_64/02676767-mailman3/builder-live.log.gz>`_.
Rebuilding the mailman3 after this failed on tests. After trying newer commits of
mailman3 to fix the failing tests and building another missing dependencies, we ended up
on the commit where we started and for which the python stack needs to be rebuild.
Before decision to cease further building attempts we built mailman3 from this `commit
<https://src.fedoraproject.org/rpms/mailman3/c/1e92c38ebbf2dea3af67c9a29f77c1d8b6c31df8?branch=rawhide>`_
in Fedora dist-git. This build has all dependencies satisfied but the `tests are failing
<https://download.copr.fedorainfracloud.org/results/zlopez/mailman3/centos-stream-8-x86_64/02676767-mailman3/builder-live.log.gz>`_.
Conclusion
----------
The build of the mailman3 in `centos-stream-8` or EPEL8 chroot will probably take few weeks to
solve and probably need providing downstream patches and newer python stack.
The build of the mailman3 in `centos-stream-8` or EPEL8 chroot will probably take few
weeks to solve and probably need providing downstream patches and newer python stack.

38
docs/mailman3/index.rst
View file

@ -4,15 +4,15 @@ Mailman3
Purpose
-------
Purpose of this investigation is to find potential ways to migrate the CentOS mailing lists to
mailman3 from the current solution and allow for a frontend. The output of the investigation
is this document.
Purpose of this investigation is to find potential ways to migrate the CentOS mailing
lists to mailman3 from the current solution and allow for a frontend. The output of the
investigation is this document.
Resources
---------
* Mailman3 upstream: https://wiki.list.org/Mailman3
* Mailman3 Fedora dist-git: https://src.fedoraproject.org/rpms/mailman3
- Mailman3 upstream: https://wiki.list.org/Mailman3
- Mailman3 Fedora dist-git: https://src.fedoraproject.org/rpms/mailman3
Investigation
-------------
@ -23,40 +23,38 @@ Here is the list of ideas/things we discussed/looked at:
:maxdepth: 1
building_rpm_centos_stream
install_using_pip_and_venv
install_using_podman
add_centos_to_fedora_server
Conclusions
-----------
Based on the investigations carried out the most beneficial but probably most difficult
solution would be to build the rpms for mailman3 hyperkitty and postorius for epel. This
would make this available for Centos but could also be used by fedora which could help
with some longstanding issues such as https://pagure.io/fedora-infrastructure/issue/5478.
with some longstanding issues such as
https://pagure.io/fedora-infrastructure/issue/5478.
The installation using containers would address the issues of compatibility and push
them upstream but it also brings the problems of relying on an upstream maintainer who
them upstream but it also brings the problems of relying on an upstream maintainer who
is not a part of our community.
When installing with pip and venv similar problems are encountered as with rpm where
When installing with pip and venv similar problems are encountered as with rpm where
dependencies are not available or up to date in the fedora or stream environment.
Adding Centos mailing list to the fedora server would be a quick fix but doubles down
on using the fedora setup which itself in not in perfect health. It would also involve
some issues with naming as the centos mail list uses centos.org rather than lists.centos.org
Adding Centos mailing list to the fedora server would be a quick fix but doubles down on
using the fedora setup which itself in not in perfect health. It would also involve some
issues with naming as the centos mail list uses centos.org rather than lists.centos.org
Proposed Roadmap
----------------
1. Package mailman3-core postorious and hyperkitty for epel. This requires python3 base
2. Deploy a server using the above packages.
3. Convert all email addresses to users. This will involve comms as emails may not
3. Convert all email addresses to users. This will involve comms as emails may not
directly relate to fas users.
3. Integrate with Accounts sso. This may potentially force mailing list users to have
an account with Fedora/Centos which was not a rule up to now.
4. Import all archives/lists. This may require an outage as any pending subsriptions or
4. Integrate with Accounts sso. This may potentially force mailing list users to have an
account with Fedora/Centos which was not a rule up to now.
5. Import all archives/lists. This may require an outage as any pending subsriptions or
emails won't be carried across.
5. Assign correct owners to lists. Currently in CentOS this is just controlles by passwords
6. Assign correct owners to lists. Currently in CentOS this is just controlles by
passwords

77
docs/mirrors-countme/index.rst
View file

@ -1,69 +1,72 @@
Improving reliability of mirrors-countme scripts
========================
================================================
Notes on curent deployment
--------------------------
For investigating and deployment, you need to be the member of
sysadmin-analysis.
For investigating and deployment, you need to be the member of sysadmin-analysis.
The repo that has the code is on https://pagure.io/mirrors-countme/
The deployment configuration is stored in ansible repo, run through playbook playbooks/groups/logserver.yml,
mostly in role roles/web-data-analysis.
The deployment configuration is stored in ansible repo, run through playbook
playbooks/groups/logserver.yml, mostly in role roles/web-data-analysis.
The scripts are running on log01.iad2.fedoraproject.org. If you are a member of sysadmin-analysis, you should be able to ssh,
and have root there.
The scripts are running on log01.iad2.fedoraproject.org. If you are a member of
sysadmin-analysis, you should be able to ssh, and have root there.
There are several cron jobs responsible for running the scripts:
* syncHttpLogs in /etc/cron.daily/ rsync logs to /var/log/hosts/$HOST/$YEAR/$MONTH/$DAY/http
* combineHttp - in /etc/cron.d/ every day at 6, runs /usr/local/bin/combineHttpLogs.sh
combines logs from /var/log/hosts to /mnt/fedora_stats/combined-http
based on the project. We are using /usr/share/awstats/tools/logresolvemerge.pl and I am not sure we are using it correctly
* condense-mirrorlogs - in /etc/cron.d/ every day at 6, does some sort of analysis, posibly one of the older scripts. It seems to attempt to sort the logs again.
* countme-update - in /etc/cron.d/ every day at 9, runs two scripts,
countme-update-rawdb.sh that parses the logs and fills in the raw database
and countme-update-totals.sh that uses the rawdb to calculate the statistics
The results of countme-update-totals.sh are then copied to a web-folder to make it available at https://data-analysis.fedoraproject.org/csv-reports/countme/
- syncHttpLogs in /etc/cron.daily/ rsync logs to
/var/log/hosts/$HOST/$YEAR/$MONTH/$DAY/http
- combineHttp - in /etc/cron.d/ every day at 6, runs /usr/local/bin/combineHttpLogs.sh
combines logs from /var/log/hosts to /mnt/fedora_stats/combined-http based on the
project. We are using /usr/share/awstats/tools/logresolvemerge.pl and I am not
sure we are using it correctly
- condense-mirrorlogs - in /etc/cron.d/ every day at 6, does some sort of analysis,
posibly one of the older scripts. It seems to attempt to sort the logs again.
- countme-update - in /etc/cron.d/ every day at 9, runs two scripts,
countme-update-rawdb.sh that parses the logs and fills in the raw database and
countme-update-totals.sh that uses the rawdb to calculate the statistics The
results of countme-update-totals.sh are then copied to a web-folder to make it
available at https://data-analysis.fedoraproject.org/csv-reports/countme/
Notes on avenues of improvement
-------------------------------
We have several areas we need to improve:
* downloading and syncing the logs, sometimes can fail or hang.
* problems when combining them
* instalation of the scripts, as there has been problem with updates,
and currently we are doing just a pull of the git repo and running the pip install
- downloading and syncing the logs, sometimes can fail or hang.
- problems when combining them
- instalation of the scripts, as there has been problem with updates, and currently we
are doing just a pull of the git repo and running the pip install
Notes on replacing with off-the shelf solutions
-----------------------------------------------
As the raw data we are basing our staticis on are just the access-logs
from our proxy-servers, we could be able to find an off-the shelf solution,
that could replace our brittle scripts.
As the raw data we are basing our staticis on are just the access-logs from our
proxy-servers, we could be able to find an off-the shelf solution, that could replace
our brittle scripts.
There are two solutions that psesent themselves, ELK stack
and Loki and Promtail by Grafana.
There are two solutions that psesent themselves, ELK stack and Loki and Promtail by
Grafana.
We are already running ELK stack on our openshift,
but our experience so far is that Elastic Search has even more brittle deployment.
We are already running ELK stack on our openshift, but our experience so far is that
Elastic Search has even more brittle deployment.
We did some experiments with Loki. The technology seems promissing,
as it is much more simple than ELK stack, with size looking comparable to the raw logs.
We did some experiments with Loki. The technology seems promissing, as it is much more
simple than ELK stack, with size looking comparable to the raw logs.
Moreover, promtail that does the parsing and uploading of logs has facilities to both
add labels to loglies that will then be indexed and queriable in the database
and collect statistics from the loglines directly that can be gathered by prometheus.
add labels to loglies that will then be indexed and queriable in the database and
collect statistics from the loglines directly that can be gathered by prometheus.
You can query the logs with language simmilar to GraphQL.
We are not going to use it because:
* it doesn't deal well with historical data, so any attempts at initial import of logsare pain.
* using promtail enabled metrics wouldn't help us with double-counting of people hitting different proxy servers
* configuration is fiddly and tricky to test
* changing batch-process to soft-realtime sounds like a headache
- it doesn't deal well with historical data, so any attempts at initial import of
logsare pain.
- using promtail enabled metrics wouldn't help us with double-counting of people hitting
different proxy servers
- configuration is fiddly and tricky to test
- changing batch-process to soft-realtime sounds like a headache

162
docs/monitoring_metrics/faq.rst
View file

@ -1,34 +1,41 @@
Frequently Asked Questions
==========================
Here are a list of questions and answers that should help you get start with
monitoring with zabbix and prometheus.
Here are a list of questions and answers that should help you get start with monitoring
with zabbix and prometheus.
How do I access zabbix?
-----------------------
1. First obtain Kerberos ticket with kinit:
::
.. code-block::
$ kinit myusername@FEDORAPROJECT.ORG
Password for myusername@FEDORAPROJECT.ORG:
2. Login to https://zabbix.stg.fedoraproject.org/zabbix.php?action=dashboard.view to see dashboard
3. If you need to be added in special privilege group (to see specific systems metrics), Open a PR in <path-to-inventory> with your FAS id in the list under the group and ask sysadmin of that groups to +1.
2. Login to https://zabbix.stg.fedoraproject.org/zabbix.php?action=dashboard.view to see
dashboard
3. If you need to be added in special privilege group (to see specific systems metrics),
Open a PR in <path-to-inventory> with your FAS id in the list under the group and ask
sysadmin of that groups to +1.
How do I access zabbix when I'm a community member?
---------------------------------------------------
1. First obtain Kerberos ticket with kinit:
::
.. code-block::
$ kinit myusername@FEDORAPROJECT.ORG
Password for myusername@FEDORAPROJECT.ORG:
2. Login to https://zabbix.stg.fedoraproject.org/zabbix.php?action=dashboard.view to see guest/public dashboard
2. Login to https://zabbix.stg.fedoraproject.org/zabbix.php?action=dashboard.view to see
guest/public dashboard
How do I access Prometheus?
---------------------------
Prometheus is running in the application monitoring namespace, standard routing applies,
i.e.: https://prometheus-route-application-monitoring.app.os.stg.fedoraproject.org/graph
@ -36,7 +43,9 @@ To access it you need to have account in the openshift it is running in.
How do I access Prometheus when I'm a community member?
-------------------------------------------------------
You shouldn't access prometheus directly, unless you are maintaining an application in openshift.
You shouldn't access prometheus directly, unless you are maintaining an application in
openshift.
Data from prometheus can be exported and viewed in Grafana or Zabbix, meaning we can
give access to a more limited public view through dashboards in one of these.
@ -44,114 +53,131 @@ give access to a more limited public view through dashboards in one of these.
Do you have a 5 minutes guide on how to use prometheus?
-------------------------------------------------------
In other words, do you have some how-tos/links I should read to understand/get
started with prometheus?
In other words, do you have some how-tos/links I should read to understand/get started
with prometheus?
* quick introduction to the stack we are running: https://www.youtube.com/watch?v=-37OPXXhrTw
* to get idea on how to use it, look at sample queries: https://prometheus.io/docs/prometheus/latest/querying/examples/
* for instrumentation, look at the libraries in https://github.com/prometheus/
- quick introduction to the stack we are running:
https://www.youtube.com/watch?v=-37OPXXhrTw
- to get idea on how to use it, look at sample queries:
https://prometheus.io/docs/prometheus/latest/querying/examples/
- for instrumentation, look at the libraries in https://github.com/prometheus/
How do I get basic HW (disk, cpu, memory, network...) monitoring for a host?
----------------------------------------------------------------------------
There are out of the box template for most of basic monitoring requirement that
can be seen on the web UI once you run the zabbix-agent-role against the node.
if you want to send any custom metrics, we recommend zabbix-sender. Zabbix sender is a command line utility that may be used to send performance data to zabbix server for processing.
Adding the zabbix sender command in crontab is one way of continuously sending
data to server that can processed on server side (in your web UI). See https://www.zabbix.com/documentation/current/manpages/zabbix_sender
There are out of the box template for most of basic monitoring requirement that can be
seen on the web UI once you run the zabbix-agent-role against the node. if you want to
send any custom metrics, we recommend zabbix-sender. Zabbix sender is a command line
utility that may be used to send performance data to zabbix server for processing.
Adding the zabbix sender command in crontab is one way of continuously sending data to
server that can processed on server side (in your web UI). See
https://www.zabbix.com/documentation/current/manpages/zabbix_sender
How do I monitor a list of services?
------------------------------------
- pagure.io and src.fp.o have two different list of services to monitor
they partly overlap but aren't exactly the same, how can I monitor them?
- For prometheus, metrics exported are usually done by instrumentation,
meaning if i.e. pagure was instrumented to export /metrics endpoint,
you just need to make sure you are collecting them, either because they run in openshift,
and you configured appropriate ServiceMonitor or PodMonitor objects,
or if outside of openshift, it is in additional scrape configuration of prometheus.
Because collected metrics are labeled, it is simple to distinguish which belong where.
- For Zabbix, if you want to send any custom metrics, we recommend zabbix-sender. Zabbix sender is a command line utility that may be used to send performance data to zabbix server for processing. Adding the zabbix sender command in crontab is one way of continuously sending data to server that can processed on server side (in your web UI). See https://www.zabbix.com/documentation/current/manpages/zabbix_sender
- pagure.io and src.fp.o have two different list of services to monitor
they partly overlap but aren't exactly the same, how can I monitor them?
- For prometheus, metrics exported are usually done by instrumentation, meaning if i.e.
pagure was instrumented to export /metrics endpoint, you just need to make sure you
are collecting them, either because they run in openshift, and you configured
appropriate ServiceMonitor or PodMonitor objects, or if outside of openshift, it is in
additional scrape configuration of prometheus. Because collected metrics are labeled,
it is simple to distinguish which belong where.
- For Zabbix, if you want to send any custom metrics, we recommend zabbix-sender. Zabbix
sender is a command line utility that may be used to send performance data to zabbix
server for processing. Adding the zabbix sender command in crontab is one way of
continuously sending data to server that can processed on server side (in your web
UI). See https://www.zabbix.com/documentation/current/manpages/zabbix_sender
How do I get alerted for a service not running?
-----------------------------------------------
- Prometheus supports configuring rules for alert-manager that can then notify through various services.
You can learn about the configuration here: https://prometheus.io/docs/alerting/latest/configuration/#configuration-file
The rules specifying when to alert are done in prometheus itself : https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
You can specify them in CRDs in your project in simmilar fashion as with Service Monitor
To use IRC, there needs to be a separate gateway installed in a sidecar: https://github.com/google/alertmanager-irc-relay
- In Zabbix, you can set custom alerting for yourself (or for groups through
web UI). Follow https://www.zabbix.com/documentation/5.0/manual/config/triggers/trigger
- Prometheus supports configuring rules for alert-manager that can then notify through
various services. You can learn about the configuration here:
https://prometheus.io/docs/alerting/latest/configuration/#configuration-file The rules
specifying when to alert are done in prometheus itself :
https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ You can
specify them in CRDs in your project in simmilar fashion as with Service Monitor To
use IRC, there needs to be a separate gateway installed in a sidecar:
https://github.com/google/alertmanager-irc-relay
- In Zabbix, you can set custom alerting for yourself (or for groups through web UI).
Follow https://www.zabbix.com/documentation/5.0/manual/config/triggers/trigger
How can I tune the alerts?
--------------------------
As in, who gets alerted? When? How?
- In Zabbix, we will have different groups with different configurations. When
you are added in that group, you will receive notifications relevant to that
group (you can change what alerting you want for the group once you have
access to that). You can filter down the alerting even more for yourself in
web UI. Follow this tutorial: https://www.zabbix.com/documentation/5.0/manual/config/triggers/trigger
If you want to tweak how you receive your alerts, follow https://www.zabbix.com/documentation/5.0/manual/config/notifications/media
- In Zabbix, we will have different groups with different configurations. When you are
added in that group, you will receive notifications relevant to that group (you can
change what alerting you want for the group once you have access to that). You can
filter down the alerting even more for yourself in web UI. Follow this tutorial:
https://www.zabbix.com/documentation/5.0/manual/config/triggers/trigger If you want to
tweak how you receive your alerts, follow
https://www.zabbix.com/documentation/5.0/manual/config/notifications/media
How do I ask for the service to be restarted <X> times before being alerted?
----------------------------------------------------------------------------
- In prometheus you can't. It is assumed you are using kubernetes that would manage something like this for you.
- In zabbix, <TODO>, you can do events based on triggers and there are event
correlation options but yet to figure out this customization
- In prometheus you can't. It is assumed you are using kubernetes that would manage
something like this for you.
- In zabbix, <TODO>, you can do events based on triggers and there are event correlation
options but yet to figure out this customization
How do I monitor rabbitmq queues?
---------------------------------
- In prometheus, according to https://www.rabbitmq.com/prometheus.html#overview-prometheus
you just need to make sure you are collecting the exported metrics.
- In Zabbix, according to https://www.zabbix.com/integrations/rabbitmq, there
is a way to build push data to zabbix that can be processed on server side
- In prometheus, according to
https://www.rabbitmq.com/prometheus.html#overview-prometheus you just need to make
sure you are collecting the exported metrics.
- In Zabbix, according to https://www.zabbix.com/integrations/rabbitmq, there is a way
to build push data to zabbix that can be processed on server side
How do we alert about checks not passing to people outside of our teams?
------------------------------------------------------------------------
-> the OSCI team is interesting in having notifications/monitoring for the CI
queues in rabbitmq
-> the OSCI team is interesting in having notifications/monitoring for the CI
queues in rabbitmq
How can we chain a prometheus instance to ours?
-----------------------------------------------
This allows to consolidate in a single instance monitoring coming from different
instances. This can be done with configuring federation in additional scrape configs: https://prometheus.io/docs/prometheus/latest/federation/
instances. This can be done with configuring federation in additional scrape configs:
https://prometheus.io/docs/prometheus/latest/federation/
How can I monitor the performances of my application?
-----------------------------------------------------
Number of requests served? Number of 500 errors? Number of DB connections?
With prometheus, you need to instrument your application and configure prometheus t collect its metrics.
With prometheus, you need to instrument your application and configure prometheus t
collect its metrics.
How do I ack an alert so it stops alerting?
-------------------------------------------
With prometheus and Alertmanager, there is no way to just ACK an alert,
it is assumed that something more high-level like opsgenie would take care of actually
interacting with regular human ops people.
With prometheus and Alertmanager, there is no way to just ACK an alert, it is assumed
that something more high-level like opsgenie would take care of actually interacting
with regular human ops people.
For small enough teams, just using silence on alert in alertmanager could be enough.
There is a sidecar that serves to provide a little bit more features to the barebones alerting.
like https://github.com/prymitive/kthxbye.
There is a sidecar that serves to provide a little bit more features to the barebones
alerting. like https://github.com/prymitive/kthxbye.
- In Zabbix, you can acknowledge the problem and it will stop alerting. Follow https://www.zabbix.com/documentation/current/manual/acknowledges
- In Zabbix, you can acknowledge the problem and it will stop alerting. Follow
https://www.zabbix.com/documentation/current/manual/acknowledges
How do I pre-emptively stop a check before I start working on an outage?
------------------------------------------------------------------------
In other words: I know that I'll cause an outage while working on <service>, how
do I turn off the checks for this service to avoid notifying admins while I'm
working on it?
In other words: I know that I'll cause an outage while working on <service>, how do I
turn off the checks for this service to avoid notifying admins while I'm working on it?
In Prometheus and Alertmanager there are Silences, where you can set a time when certain alerts wouldn't
be firing. You are able to create and remove these through rest api,
In Prometheus and Alertmanager there are Silences, where you can set a time when certain
alerts wouldn't be firing. You are able to create and remove these through rest api,
- In Zabbix, simplest way is to stop zabbix agent (or custom sender) on the system and ack on
server side that it's not reachable.
- In Zabbix, simplest way is to stop zabbix agent (or custom sender) on the system and
ack on server side that it's not reachable.

53
docs/monitoring_metrics/index.rst
View file

@ -1,53 +1,62 @@
Monitoring / Metrics
========================
====================
As an ARC team initiative we want to investigate Prometheus and Zabbix
as our new monitoring and metrics solutions, by:
As an ARC team initiative we want to investigate Prometheus and Zabbix as our new
monitoring and metrics solutions, by:
- Installing Zabbix server in a VM, and hooking up the staging dist-git to it with an agent
- Installing Prometheus in our Open Shift and collecting metrics for a selected project in a self-service fashion
- Installing Zabbix server in a VM, and hooking up the staging dist-git to it with
an agent
- Installing Prometheus in our Open Shift and collecting metrics for a selected
project in a self-service fashion
Prior POCs/deployments
----------------------
Fabian Arrotin deployed and utilizes zabbix in centos infrastructure.
- https://github.com/CentOS/ansible-role-zabbix-server
- https://github.com/CentOS/ansible-role-zabbix-server
Adam Saleh has deployed a POC prometheus deployment for CoreOS team.
- https://pagure.io/centos-infra/issue/112
- https://pagure.io/centos-infra/issue/112
David Kirwan was part of the development team of https://github.com/integr8ly/application-monitoring-operator/ and did some POC around prometheus push-gateway in centos openshift
David Kirwan was part of the development team of
https://github.com/integr8ly/application-monitoring-operator/ and did some POC around
prometheus push-gateway in centos openshift
Investigation
-------------
In process we want to be able to answer the questions posed in the latest mailing thread and by the end have a setup that can lead directly into mirating us away from nagios. The questions (mostly from Kevin):
In process we want to be able to answer the questions posed in the latest mailing thread
and by the end have a setup that can lead directly into mirating us away from nagios.
The questions (mostly from Kevin):
- How can we provision both of them automatically from ansible?
- Can we get zabbix to pull from prometheus?
- Can zabbix handle our number of machines?
- How flexible is the alerting?
- How can we provision both of them automatically from ansible?
- Can we get zabbix to pull from prometheus?
- Can zabbix handle our number of machines?
- How flexible is the alerting?
Main takeaway
-------------
We managed to create proof-of-concept monitoring solutions with both prometheus and zabbix.
We managed to create proof-of-concept monitoring solutions with both prometheus and
zabbix.
The initial configuration has proven to have more pitfals than expected,
with Prometheus especially in the integration with openshift and its other auxialiary services,
and with Zabbix espcially with correctly setting up the ip-tables and network permissions,
and with configuring a reasonable setup for the user-access and user-account management.
The initial configuration has proven to have more pitfals than expected, with Prometheus
especially in the integration with openshift and its other auxialiary services, and with
Zabbix espcially with correctly setting up the ip-tables and network permissions, and
with configuring a reasonable setup for the user-access and user-account management.
Even despite these setbacks, we still feel this would be an improvement over our current setup based on Nagios.
Even despite these setbacks, we still feel this would be an improvement over our current
setup based on Nagios.
To get a basic overview of Prometheus, you can watch this short tech-talk by Adam Saleh:
(accessible only to RedHat) https://drive.google.com/file/d/1-uEIkS2jaJ2b8V_4y-AKW1J6sdZzzlc9/view
or read up the more indepth report in the relevant sections of this documentation.
(accessible only to RedHat)
https://drive.google.com/file/d/1-uEIkS2jaJ2b8V_4y-AKW1J6sdZzzlc9/view or read up the
more indepth report in the relevant sections of this documentation.
.. toctree::
:maxdepth: 1
prometheus_for_ops
prometheus_for_dev
zabbix
faq

131
docs/monitoring_metrics/prometheus_for_dev.rst
View file

@ -1,96 +1,101 @@
Notes on application monitoring self-service
---------------------------------
============================================
To get the application monitored in the given namespace, the namespace must have the correct label applied,
an in the namespace there needs to be either PodMonitor or ServiceMonitor CRD setup,
that points towards the service or pod that exports metrics.
To get the application monitored in the given namespace, the namespace must have the
correct label applied, an in the namespace there needs to be either PodMonitor or
ServiceMonitor CRD setup, that points towards the service or pod that exports metrics.
This way, the merics will be scraped into the configured prometheus and correctly labeled.
This way, the merics will be scraped into the configured prometheus and correctly
labeled.
As an example, lets look at ServiceMonitor for bodhi:
::
.. code-block::
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
monitoring-key: cpe
name: bodhi-service
namespace: bodhi
spec:
endpoints:
- path: /metrics
selector:
matchLabels:
service: web
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
monitoring-key: cpe
name: bodhi-service
namespace: bodhi
spec:
endpoints:
- path: /metrics
selector:
matchLabels:
service: web
In this example, we are only targetting the service wit label service:web, but we have the entire matching
machinery at our disposal, see `Matcher <https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#labelselector-v1-meta>`_ .
In this example, we are only targetting the service wit label service:web, but we have
the entire matching machinery at our disposal, see `Matcher
<https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#labelselector-v1-meta>`_
.
To manage alerting, you can create an alerting rule:
::
.. code-block::
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
monitoring-key: cpe
name: bodhi-rules
spec:
spec:
groups:
- name: general.rules
rules:
- alert: DeadMansSwitch
annotations:
description: >-
This is a DeadMansSwitch meant to ensure that the entire Alerting
pipeline is functional.
summary: Alerting DeadMansSwitch
expr: vector(1)
labels:
severity: none
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
monitoring-key: cpe
name: bodhi-rules
spec:
spec:
groups:
- name: general.rules
rules:
- alert: DeadMansSwitch
annotations:
description: >-
This is a DeadMansSwitch meant to ensure that the entire Alerting
pipeline is functional.
summary: Alerting DeadMansSwitch
expr: vector(1)
labels:
severity: none
This would create a alert, that will always fire, to serve as a check the alerting works.
You should be able to see it in alert manager.
This would create a alert, that will always fire, to serve as a check the alerting
works. You should be able to see it in alert manager.
To have an alert that actually does something, you should set expr to something else than vector(1).
For example, to alert on rate of 500 responses of a service going over 5/s in past 10 minutes:
To have an alert that actually does something, you should set expr to something else
than vector(1). For example, to alert on rate of 500 responses of a service going over
5/s in past 10 minutes:
sum(rate(pyramid_request_count{job="bodhi-web", status="500"}[10m])) > 5
The alerts themselves would be the routed for further processing and notification according to rules in alertmanager,
these are not available to change from the developers namespaces.
The alerts themselves would be the routed for further processing and notification
according to rules in alertmanager, these are not available to change from the
developers namespaces.
The managing and acknowledging of the alerts can be done in alert-manager in rudimentary fashion.
The managing and acknowledging of the alerts can be done in alert-manager in rudimentary
fashion.
Notes on instrumenting the application
--------------------------------------
======================================
Prometheus expects applications to scrape metrics from
to be services, with '/metrics' endpoint exposed with metrics in correct
format.
Prometheus expects applications to scrape metrics from to be services, with '/metrics'
endpoint exposed with metrics in correct format.
There are libraries that help with this for many different languages,
confusingly called client-libraries, eve though they usually export metrics as a http server endpoint:
There are libraries that help with this for many different languages, confusingly called
client-libraries, eve though they usually export metrics as a http server endpoint:
https://prometheus.io/docs/instrumenting/clientlibs/
As part of the proof of concept we have instrumented Bodhi application,
to collect data through prometheus_client python library:
As part of the proof of concept we have instrumented Bodhi application, to collect data
through prometheus_client python library:
https://github.com/fedora-infra/bodhi/pull/4079
Notes on alerting
-----------------
=================
To be be notified of alerts, you need to be subscribed to recievers that
have been configured in alertmanager.
To be be notified of alerts, you need to be subscribed to recievers that have been
configured in alertmanager.
The configuration of the rules you want to alert on can be done in the namspace of your application.
For example:
The configuration of the rules you want to alert on can be done in the namspace of your
application. For example:
::
.. code-block::
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule

237
docs/monitoring_metrics/prometheus_for_ops.rst
View file

@ -1,80 +1,97 @@
Monitoring / Metrics with Prometheus
========================
====================================
For deployment, we used combination for configuration of prometheus operator and application-monitoring operator.
For deployment, we used combination for configuration of prometheus operator and
application-monitoring operator.
Beware, most of the deployment notes could be mostly obsolete in really short time.
The POC was done on OpenShift 3.11, which limited us in using older version of prometheus operator,
as well as the no longer maintained application-monitoring operator.
Beware, most of the deployment notes could be mostly obsolete in really short time. The
POC was done on OpenShift 3.11, which limited us in using older version of prometheus
operator, as well as the no longer maintained application-monitoring operator.
In openshift 4.x that we plan to use in the near future, there is supported way integrated in the openshift deployment:
In openshift 4.x that we plan to use in the near future, there is supported way
integrated in the openshift deployment:
* https://docs.openshift.com/container-platform/4.7/monitoring/understanding-the-monitoring-stack.html
* https://docs.openshift.com/container-platform/4.7/monitoring/configuring-the-monitoring-stack.html#configuring-the-monitoring-stack
* https://docs.openshift.com/container-platform/4.7/monitoring/enabling-monitoring-for-user-defined-projects.html
The supported stack is more limited, especially w.r.t. adding user defined pod- and service-monitors, but even if we would want to
run additional prometheus instances, we should be able to skip the instalation of the necessary operators, as all of them should already be present.
- https://docs.openshift.com/container-platform/4.7/monitoring/understanding-the-monitoring-stack.html
- https://docs.openshift.com/container-platform/4.7/monitoring/configuring-the-monitoring-stack.html#configuring-the-monitoring-stack
- https://docs.openshift.com/container-platform/4.7/monitoring/enabling-monitoring-for-user-defined-projects.html
The supported stack is more limited, especially w.r.t. adding user defined pod- and
service-monitors, but even if we would want to run additional prometheus instances, we
should be able to skip the instalation of the necessary operators, as all of them should
already be present.
Notes on operator deployment
-------------------
----------------------------
Operator pattern is often used with kubernetes and openshift for more complex deployments.
Instead of applying all of the configuration to deploy your services, you deploy a special,
smaller service called operator, that has necessary permissions to deploy and configure the complex service.
Once the operator is running, instead of configuring the service itself with service-specific config-maps,
you create operator specific kubernetes objects, so-alled CRDs.
Operator pattern is often used with kubernetes and openshift for more complex
deployments. Instead of applying all of the configuration to deploy your services, you
deploy a special, smaller service called operator, that has necessary permissions to
deploy and configure the complex service. Once the operator is running, instead of
configuring the service itself with service-specific config-maps, you create operator
specific kubernetes objects, so-alled CRDs.
The deployment of the operator in question was done by configuring the CRDs, roles and rolebinding and operator setup:
The deployment of the operator in question was done by configuring the CRDs, roles and
rolebinding and operator setup:
The definitions are as follows:
- https://github.com/prometheus-operator/prometheus-operator/tree/v0.38.3/example/prometheus-operator-crd
- https://github.com/prometheus-operator/prometheus-operator/tree/v0.38.3/example/rbac/prometheus-operator-crd
- https://github.com/prometheus-operator/prometheus-operator/tree/v0.38.3/example/rbac/prometheus-operator
The definitions are as follows: -
https://github.com/prometheus-operator/prometheus-operator/tree/v0.38.3/example/prometheus-operator-crd
-
https://github.com/prometheus-operator/prometheus-operator/tree/v0.38.3/example/rbac/prometheus-operator-crd
-
https://github.com/prometheus-operator/prometheus-operator/tree/v0.38.3/example/rbac/prometheus-operator
Once the operator is correctly running, you just define a prometheus crd and it will create prometheus deployment for you.
The POC lives in https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/application-monitoring.yml
Once the operator is correctly running, you just define a prometheus crd and it will
create prometheus deployment for you.
The POC lives in
https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/application-monitoring.yml
Notes on application monitoring operator deployment
---------------------------------------------------
The application-monitoring operator was created to solve the integration of Prometheus, Alertmanager and Grafana.
After you configure it, it configures the relevant operators responsible for these services.
The application-monitoring operator was created to solve the integration of Prometheus,
Alertmanager and Grafana. After you configure it, it configures the relevant operators
responsible for these services.
The most interesting difference between configuring this shared operator,
compared to configuring these operators individually is that it configures some of the integrations,
The most interesting difference between configuring this shared operator, compared to
configuring these operators individually is that it configures some of the integrations,
and it integrates well with openshifts auth system through oauth proxy.
The biggest drawback is, that the application-monitoring operator is orphanned project,
but because it mostly configures other operators, it is relatively simple to just recreate
the configuration for both prometheus and alertmanager to be deployed,
and deploy the prometheus and alertmanager operators without the help or the application-monitoring operator.
but because it mostly configures other operators, it is relatively simple to just
recreate the configuration for both prometheus and alertmanager to be deployed, and
deploy the prometheus and alertmanager operators without the help or the
application-monitoring operator.
Notes on persistence
--------------------
Prometheus by default expects to have a writable /prometheus folder,
that can serve as persistent storage.
Prometheus by default expects to have a writable /prometheus folder, that can serve as
persistent storage.
For the persistent volume to work for this purpose, it has to
**needs to have POSIX-compliant filesystem**, and NFS we currently have configured is not.
This is discussed in the `operational aspects <https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects>`_
of Prometheus documentation
For the persistent volume to work for this purpose, it has to **needs to have
POSIX-compliant filesystem**, and NFS we currently have configured is not. This is
discussed in the `operational aspects
<https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects>`_ of
Prometheus documentation
The easiest supported way to have a POSIX-compliant `filesystem is to setup local-storage <https://docs.openshift.com/container-platform/3.11/install_config/configuring_local.html>`_
The easiest supported way to have a POSIX-compliant `filesystem is to setup
local-storage
<https://docs.openshift.com/container-platform/3.11/install_config/configuring_local.html>`_
in the cluster.
In 4.x versions of OpenShift `there is a local-storage-operator <https://docs.openshift.com/container-platform/4.7/storage/persistent_storage/persistent-storage-local.html>`_ for this purpose.
In 4.x versions of OpenShift `there is a local-storage-operator
<https://docs.openshift.com/container-platform/4.7/storage/persistent_storage/persistent-storage-local.html>`_
for this purpose.
This is the simplest way to have working persistence, but it prevents us to have multiple instanes
across openshift nodes, as the pod is using the underlying filesystem on the node.
This is the simplest way to have working persistence, but it prevents us to have
multiple instanes across openshift nodes, as the pod is using the underlying filesystem
on the node.
To ask the operator to create persisted prometheus, you specify in its configuration i.e.:
To ask the operator to create persisted prometheus, you specify in its configuration
i.e.:
::
.. code-block::
storage:
volumeClaimTemplate:
@ -87,27 +104,27 @@ To ask the operator to create persisted prometheus, you specify in its configura
By default retention is set to 24 hours and can be over-ridden
Notes on long term storage
--------------------
--------------------------
Usually, prometheus itself is setup to store its metrics for shorter ammount of time,
and it is expected that for longterm storage and analysis, there is some other storage solution,
such as influxdb or timescaledb.
and it is expected that for longterm storage and analysis, there is some other storage
solution, such as influxdb or timescaledb.
We are currently running a POC that sychronizes Prometheus with Timescaledb (running on Postgresql)
through a middleware service called `promscale <https://github.com/timescale/promscale>`_ .
We are currently running a POC that sychronizes Prometheus with Timescaledb (running on
Postgresql) through a middleware service called `promscale
<https://github.com/timescale/promscale>`_ .
Promscale just needs an access to a appropriate postgresql database:
and can be configured through PROMSCALE_DB_PASSWORD, PROMSCALE_DB_HOST.
Promscale just needs an access to a appropriate postgresql database: and can be
configured through PROMSCALE_DB_PASSWORD, PROMSCALE_DB_HOST.
By default it will ensure the database has timescaledb installed and configures its database
automatically.
By default it will ensure the database has timescaledb installed and configures its
database automatically.
We setup prometheus with directive to use promscale service as a backend:
https://github.com/timescale/promscale
::
.. code-block::
remote_write:
- url: "http://promscale:9201/write"
@ -117,24 +134,27 @@ https://github.com/timescale/promscale
Notes on auxialiary services
----------------------------
As prometheus is primarily targeted to collect metrics from
services that have beein instrumented to expose them, if your service is not instrumented,
or it is not a service, i.e. a batch-job, you need an adapter to help you with the metrics collection.
As prometheus is primarily targeted to collect metrics from services that have beein
instrumented to expose them, if your service is not instrumented, or it is not a
service, i.e. a batch-job, you need an adapter to help you with the metrics collection.
There are two services that help with this.
* `blackbox exporter <https://github.com/prometheus/blackbox_exporter>`_ to monitor services that have not been instrumented based on querying public a.p.i.
* `push gateway <https://prometheus.io/docs/practices/pushing/#should-i-be-using-the-pushgateway>`_ that helps collect information from batch-jobs
- `blackbox exporter <https://github.com/prometheus/blackbox_exporter>`_ to monitor
services that have not been instrumented based on querying public a.p.i.
- `push gateway
<https://prometheus.io/docs/practices/pushing/#should-i-be-using-the-pushgateway>`_
that helps collect information from batch-jobs
Maintaining the push-gateway can be relegated to the application developer,
as it is lightweight, and by colloecting metrics from the namespace it is running in,
the data will be correctly labeled.
Maintaining the push-gateway can be relegated to the application developer, as it is
lightweight, and by colloecting metrics from the namespace it is running in, the data
will be correctly labeled.
With blackbox exporter, it can be beneficial to have it running as prometheus side-car,
in simmilar fashion, as we configure oauth-proxy, adding this to the containers section
of the prometheus definition:
::
.. code-block::
- name: blackbox-exporter
volumeMounts:
@ -149,58 +169,65 @@ of the prometheus definition:
- containerPort: 9115
name: blackbox
We can then instruct what is to be monitored through the configmap-blackbox, you can find `relevant examples <https://github.com/prometheus/blackbox_exporter/blob/master/example.yml>` in the project repo.
Beause blackox exporter is in the same pod, we need to use the additional-scrape-config to add it in.
We can then instruct what is to be monitored through the configmap-blackbox, you can
find `relevant examples
<https://github.com/prometheus/blackbox_exporter/blob/master/example.yml>` in the
project repo. Beause blackox exporter is in the same pod, we need to use the
additional-scrape-config to add it in.
Notes on alerting
-----------------
Prometheus as is, can have rules configured that trigger alerts, once
a specific query evaluates to true. The definition of the rule is explained in the companion docs
for prometheus for developers and can be created in the namespace of the running application.
Prometheus as is, can have rules configured that trigger alerts, once a specific query
evaluates to true. The definition of the rule is explained in the companion docs for
prometheus for developers and can be created in the namespace of the running
application.
Here, we need to focus what happens with alert after prometheus realizes it should fire it,
based on a rule.
Here, we need to focus what happens with alert after prometheus realizes it should fire
it, based on a rule.
In prometheus crd definition, there is a section about the alert-manager that is supposed to
manage the forwarding of these alerts.
In prometheus crd definition, there is a section about the alert-manager that is
supposed to manage the forwarding of these alerts.
::
.. code-block::
alerting:
alertmanagers:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
name: alertmanager-service
namespace: application-monitoring
port: web
scheme: https
tlsConfig:
caFile: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
serverName: alertmanager-service.application-monitoring.svc
alerting:
alertmanagers:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
name: alertmanager-service
namespace: application-monitoring
port: web
scheme: https
tlsConfig:
caFile: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
serverName: alertmanager-service.application-monitoring.svc
We already have alertmanager running and configured by the alertmanager-operator.
Alertmanager itself is really simplistic with a simple ui and api, that allows for silencing an
alert for a given ammount of time.
Alertmanager itself is really simplistic with a simple ui and api, that allows for
silencing an alert for a given ammount of time.
It is expected that the actual user-interaction is happening elsewhere,
either through services like OpsGenie, or through i.e. `integration with zabbix <https://devopy.io/setting-up-zabbix-alertmanager-integration/>`_
It is expected that the actual user-interaction is happening elsewhere, either through
services like OpsGenie, or through i.e. `integration with zabbix
<https://devopy.io/setting-up-zabbix-alertmanager-integration/>`_
More of a build-it yourself solution is to use i.e. https://karma-dashboard.io/,
but we haven't tried any of these as the part of our POC.
More of a build-it yourself solution is to use i.e. https://karma-dashboard.io/, but we
haven't tried any of these as the part of our POC.
To be able to be notified of the alert, you need to have the `correct reciever configuration <https://prometheus.io/docs/alerting/latest/configuration/#email_config>`_ in the alertmanagers secret:
To be able to be notified of the alert, you need to have the `correct reciever
configuration <https://prometheus.io/docs/alerting/latest/configuration/#email_config>`_
in the alertmanagers secret:
::
.. code-block::
global:
resolve_timeout: 5m
route:
group_by: ['job']
group_wait: 10s
group_interval: 10s
repeat_interval: 30m
receiver: 'email'
receivers:
- name: 'email'
email_configs:
- to: 'asaleh@redhat.com'
global:
resolve_timeout: 5m
route:
group_by: ['job']
group_wait: 10s
group_interval: 10s
repeat_interval: 30m
receiver: 'email'
receivers:
- name: 'email'
email_configs:
- to: 'asaleh@redhat.com'

43
docs/monitoring_metrics/zabbix.rst
View file

@ -1,38 +1,39 @@
Monitoring / Metrics with Prometheus
========================
====================================
We are using Zabbix 5.0 (lts) server with PostgreSQL database. Starting with manual configuration in a test vm and then automating it for for deployment, Ansible roles `zabbix-server` and `zabbix-agent` are to results of this PoC work.
Please follow FAQ to see how to access staging deployment of zabbix.
We are using Zabbix 5.0 (lts) server with PostgreSQL database. Starting with manual
configuration in a test vm and then automating it for for deployment, Ansible roles
`zabbix-server` and `zabbix-agent` are to results of this PoC work. Please follow FAQ to
see how to access staging deployment of zabbix.
zabbix-server
-------------
This role is ready at the base level but as the complexity of the monitoring
increases, more work would be needed. At the current level, it
This role is ready at the base level but as the complexity of the monitoring increases,
more work would be needed. At the current level, it
* Installs needed packages for server
* configure zabbix, apache and PostgreSQL configuration files
* configures web UI
* configures kerberos authentication
- Installs needed packages for server
- configure zabbix, apache and PostgreSQL configuration files
- configures web UI
- configures kerberos authentication
While these basic things are good for POC, they are not ready to be in
production until we have configured the following
While these basic things are good for POC, they are not ready to be in production until
we have configured the following
* add inventory files for groups and users and have zabbix-cli restore those
in case of a fresh installation
* Network config audit (see common challenges)
- add inventory files for groups and users and have zabbix-cli restore those in case
of a fresh installation
- Network config audit (see common challenges)
zabbix-agent
------------
This role is ready to be used and existing templates are good to gather basic
information. Though specific of what kind of common data would be collected
from all agent nodes needs to be discussed widely and set in template.
Other than common metrics, one can also export custom metrics using
zabbix-sender (see FAQ).
information. Though specific of what kind of common data would be collected from all
agent nodes needs to be discussed widely and set in template. Other than common metrics,
one can also export custom metrics using zabbix-sender (see FAQ).
Common challenges
-----------------
Lack of experience in selinux policies and network configuration, we are not
very confident with those. A veteran sysadmin would be needed audit.
Lack of experience in selinux policies and network configuration, we are not very
confident with those. A veteran sysadmin would be needed audit.

76
docs/package-automation/index.rst
View file

@ -4,26 +4,30 @@ Package Automation
Purpose
-------
Not all Fedora applications are published as RPMs or are not always up to date to their latest code release, automating this workflow would fix that.
Not all Fedora applications are published as RPMs or are not always up to date to their
latest code release, automating this workflow would fix that.
Other teams would benefit from this work, such as those that build application containers or running integration tests based on RPM releases.
Other teams would benefit from this work, such as those that build application
containers or running integration tests based on RPM releases.
Resources
---------
* Packit: https://packit.dev
* Bodhi: https://bodhi.fedoraproject.org/docs/
* Packit Pagure support: https://github.com/packit/packit-service/issues/556
- Packit: https://packit.dev
- Bodhi: https://bodhi.fedoraproject.org/docs/
- Packit Pagure support: https://github.com/packit/packit-service/issues/556
Investigation
-------------
The investigation was about testing several tools to accomplish this: Github actions, Jenkins, Packit and Bodhi.
The investigation was about testing several tools to accomplish this: Github actions,
Jenkins, Packit and Bodhi.
Packit does provide a great level of abstraction to both build and publish a package update, integrating with Bodhi and Koji in the way.
Packit does provide a great level of abstraction to both build and publish a package
update, integrating with Bodhi and Koji in the way.
Packit is also available in both Github and Gitlab as a CI application but not in Pagure (there is an open issue in their repository for implementing it).
Packit is also available in both Github and Gitlab as a CI application but not in Pagure
(there is an open issue in their repository for implementing it).
Here is the list of ideas/things we discussed/looked at:
@ -34,64 +38,70 @@ Here is the list of ideas/things we discussed/looked at:
packit-triggers
packit-and-pagure
Conclusions
-----------
Packit looks to be the path to take, specially because it can integrate with different tools, such as COPR and Bodhi.
Packit looks to be the path to take, specially because it can integrate with different
tools, such as COPR and Bodhi.
There is an open questions as how to deal with apps that are hosted in Pagure which are described in the "Proposed Roadmap" section.
There is an open questions as how to deal with apps that are hosted in Pagure which are
described in the "Proposed Roadmap" section.
Proposed Roadmap
----------------
We propose the use of Packit CI Server, usually refered as Packit Service, because it can integrate with both Koji, COPR and Bodhi - it also means that
we don't need to maintain our own CI service to use packit.
We propose the use of Packit CI Server, usually refered as Packit Service, because it
can integrate with both Koji, COPR and Bodhi - it also means that we don't need to
maintain our own CI service to use packit.
Packit CI is also available in Gitlab which means the migration to other SCMs will have minimal impact on this work.
Packit CI is also available in Gitlab which means the migration to other SCMs will have
minimal impact on this work.
Github
------
* Enable/Install Packit Github app: https://packit.dev/docs/packit-service/#integrating-packit-as-a-service-into-your-project-or-organization-from-github-marketplace;
* Add a `.packit.yaml` file in each repository (can be done with `packit init`);
* Sample: https://github.com/fedora-infra/the-new-hotness/pull/432
- Enable/Install Packit Github app:
https://packit.dev/docs/packit-service/#integrating-packit-as-a-service-into-your-project-or-organization-from-github-marketplace;
- Add a `.packit.yaml` file in each repository (can be done with `packit init`);
- Sample: https://github.com/fedora-infra/the-new-hotness/pull/432
Pagure
------
Pagure is not supported in Packit at the time of this writing.
There is an open issue on the Packit team side to enable this in Pagure: https://github.com/packit/packit-service/issues/556.
There is an open issue on the Packit team side to enable this in Pagure:
https://github.com/packit/packit-service/issues/556.
This could be an opportunity for CPE to work with the Packit team
to enable this in pagure, but that will require development effort on our side.
This could be an opportunity for CPE to work with the Packit team to enable this in
pagure, but that will require development effort on our side.
We could split the initiative in two phases:
* Phase 1: enable Packit in apps hosted in Github;
* Phase 2: Support Packit in Pagure and enable apps in Pagure to use Packit.
- Phase 1: enable Packit in apps hosted in Github;
- Phase 2: Support Packit in Pagure and enable apps in Pagure to use Packit.
Those do not depend on each other and could be worked in parallel but that would mean we need to dedicated more engineers to the initiative.
Those do not depend on each other and could be worked in parallel but that would mean we
need to dedicated more engineers to the initiative.
Team
----
* A team of three engineers would be more than enough;
* One experienced engineer to lead the team.
- A team of three engineers would be more than enough;
- One experienced engineer to lead the team.
Skills
------
Github Apps
***********
~~~~~~~~~~~
* Git;
* YAML;
* Basic packit knowdlege;
* Basic build tools knowdlege (bodhi, koji, copr, etc);
- Git;
- YAML;
- Basic packit knowdlege;
- Basic build tools knowdlege (bodhi, koji, copr, etc);
Packit + Pagure
***************
~~~~~~~~~~~~~~~
* All of the previous section + Python knowledge.
- All of the previous section + Python knowledge.

25
docs/package-automation/packit-and-pagure.rst
View file

@ -2,30 +2,35 @@ Packit and Pagure
=================
Resources
----------
---------
* https://github.com/packit/packit-service/issues/556
- https://github.com/packit/packit-service/issues/556
Discussion
----------
This discussion is about enabling Pagure support in Packit, what needs to be done and the amount of effort to do so.
This discussion is about enabling Pagure support in Packit, what needs to be done and
the amount of effort to do so.
What would be needed
--------------------
Packit team estimates that would be 2-3 months implementation time but prioritization on their side is needed.
Packit team estimates that would be 2-3 months implementation time but prioritization on
their side is needed.
The CPE team could contribute to this integration as the Packit codebase is structured in a way that a "Pagure Handler"
can be worked in parallel with other Packit code changes.
The CPE team could contribute to this integration as the Packit codebase is structured
in a way that a "Pagure Handler" can be worked in parallel with other Packit code
changes.
Packit codebase is written in Python: https://github.com/packit/packit-service.
Conclusion
----------
The CPE could work on this and enable Pagure support, if we really want to invest in Pagure support.
The CPE could work on this and enable Pagure support, if we really want to invest in
Pagure support.
The topic of centralizing events in fedora-messaging for all SCMs showed up during this discussion.
there were some talks about it but nobody never took the time to work/design it - it is whole new topic that
should not be mixed in this work but something to consider for a future initiative.
The topic of centralizing events in fedora-messaging for all SCMs showed up during this
discussion. there were some talks about it but nobody never took the time to work/design
it - it is whole new topic that should not be mixed in this work but something to
consider for a future initiative.

26
docs/package-automation/packit-triggers.rst
View file

@ -2,9 +2,9 @@ Packit Triggers
===============
Resources
----------
---------
* Packit Triggers: https://packit.dev/docs/configuration/#packit-service-jobs
- Packit Triggers: https://packit.dev/docs/configuration/#packit-service-jobs
Discussion
----------
@ -13,17 +13,20 @@ The discussion was about when and where to push package changes.
Packit has three event triggers:
* Pull Requests
* Commits
* Releases
- Pull Requests
- Commits
- Releases
The topic showed up because pushing package changes on every and single Pull Requests sound unreasonable in some cases.
The topic showed up because pushing package changes on every and single Pull Requests
sound unreasonable in some cases.
What would be needed
--------------------
* PR CI testing is done using COPR, we can centralize those builds in a specific repository if we want to;
* Use feature branches to automatically push updates to specific Koji tags (manual releases or commits);
- PR CI testing is done using COPR, we can centralize those builds in a specific
repository if we want to;
- Use feature branches to automatically push updates to specific Koji tags (manual
releases or commits);
Everything is configured in the packit yaml file.
@ -32,6 +35,7 @@ Conclusion
The conclusion is that we all triggers for different scenarios:
* Pull Requests triggers using COPR for CI RPM test;
* Merging of the main branch can push updates to rawhide;
* Github releases and/or commits to feature branches could trigger updates in "production koji targets".
- Pull Requests triggers using COPR for CI RPM test;
- Merging of the main branch can push updates to rawhide;
- Github releases and/or commits to feature branches could trigger updates in
"production koji targets".

48
docs/package-automation/the-ci-platform.rst
View file

@ -2,45 +2,49 @@ The CI Platform
===============
Resources
----------
* Packit + Github: https://packit.dev/docs/packit-service/#integrating-packit-as-a-service-into-your-project-or-organization-from-github-marketplace
* Packit + Gitlab: https://packit.dev/docs/faq/#can-we-use-packit-with-gitlab
---------
- Packit + Github:
https://packit.dev/docs/packit-service/#integrating-packit-as-a-service-into-your-project-or-organization-from-github-marketplace
- Packit + Gitlab: https://packit.dev/docs/faq/#can-we-use-packit-with-gitlab
Discussion
----------
This discussion was about which CI platform to use, self-hosted (Jenkins/Zuul), Packit, Github actions etc.
This discussion was about which CI platform to use, self-hosted (Jenkins/Zuul), Packit,
Github actions etc.
What would be needed
--------------------
Github
******
~~~~~~
* Add a `.packit.yaml` file in each repository (those can be created by running `packit init`);
* A spec file in each repository;
* Organization needs to have packit enabled (we already do).
* Around 31 repositories in total but we are not sure how many of those we can skip.
- Add a `.packit.yaml` file in each repository (those can be created by running `packit
init`);
- A spec file in each repository;
- Organization needs to have packit enabled (we already do).
- Around 31 repositories in total but we are not sure how many of those we can skip.
Pagure
******
~~~~~~
There is no Pagure support in Packit at this moment, which means we need some workarounds to make it happen:
There is no Pagure support in Packit at this moment, which means we need some
workarounds to make it happen:
* Create mirrors in Github for those repositories;
* Use Jenkins/Zuul to run packit commands;
* Ignore it until we move those apps to something else.
- Create mirrors in Github for those repositories;
- Use Jenkins/Zuul to run packit commands;
- Ignore it until we move those apps to something else.
Conclusion
----------
Packit is the right choice for us:
Packit is the right choice for us:
* Packit is available as a "CI Plataform" in both Github and Gitlab;
* It supports multiple "build-systems", such as Koji and COPR and it also supports Bodhi for package updates;
* All we need is to enable Packit in groups/organizations (already enabled in fedora-infra Github org) and a yaml file;
* It has a CLI which enables us to re-produce its behavior locally;
* It is not maintained by CPE but the team is close enough to us.
- Packit is available as a "CI Plataform" in both Github and Gitlab;
- It supports multiple "build-systems", such as Koji and COPR and it also supports Bodhi
for package updates;
- All we need is to enable Packit in groups/organizations (already enabled in
fedora-infra Github org) and a yaml file;
- It has a CLI which enables us to re-produce its behavior locally;
- It is not maintained by CPE but the team is close enough to us.

143
docs/pagure2gitlab/gitlab.rst
View file

@ -3,49 +3,50 @@
GitLab API investigation
========================
This document investigates API calls that are needed to address the requirements for Pagure to GitLab
importer. Currently the GitLab provides both `REST <https://docs.gitlab.com/ee/api/rest/>`_
and `GraphQL <https://docs.gitlab.com/ee/api/graphql/>`_ APIs. In this investigation I will only
focus on REST API v4, because GraphQL API doesn't provide required calls
(importing project and importing merge requests).
This document investigates API calls that are needed to address the requirements for
Pagure to GitLab importer. Currently the GitLab provides both `REST
<https://docs.gitlab.com/ee/api/rest/>`_ and `GraphQL
<https://docs.gitlab.com/ee/api/graphql/>`_ APIs. In this investigation I will only
focus on REST API v4, because GraphQL API doesn't provide required calls (importing
project and importing merge requests).
All the REST API calls need user to provide
`GitLab API token <https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html>`_.
All the REST API calls need user to provide `GitLab API token
<https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html>`_.
For the purpose of this investigation I tried to import `ARC repository
<https://pagure.io/fedora-infra/arc>`_ to GitLab.
GitLab API documentation: https://docs.gitlab.com/ee/api/
Importing a git repository
--------------------------
Project can be imported by using `create project call
<https://docs.gitlab.com/ee/api/projects.html#create-project>`_. Following is a payload for import
of the ARC repository.
<https://docs.gitlab.com/ee/api/projects.html#create-project>`_. Following is a payload
for import of the ARC repository.
.. code-block:: json
{
"name":"arc",
"description":"The Advance Reconnaissance Crew",
"namespace_id":"10910066",
"import_url":"https://pagure.io/fedora-infra/arc.git"
}
This creates `ARC repository on GitLab <https://gitlab.com/testgroup519/arc>`_ with the whole
commit history and users mapped by the e-mail.
{
"name":"arc",
"description":"The Advance Reconnaissance Crew",
"namespace_id":"10910066",
"import_url":"https://pagure.io/fedora-infra/arc.git"
}
This creates `ARC repository on GitLab <https://gitlab.com/testgroup519/arc>`_ with the
whole commit history and users mapped by the e-mail.
Importing ticket
----------------
Ticket can be imported by `creating an issue <https://docs.gitlab.com/ee/api/issues.html#new-issue>`_
, `commenting on it <https://docs.gitlab.com/ee/api/notes.html#create-new-issue-note>`_,
eventually close it by `editing the issue
<https://docs.gitlab.com/ee/api/notes.html#create-new-issue-note>`_ and add any attachments by
`uploading file to project <https://docs.gitlab.com/ee/api/projects.html#upload-a-file>`_.
Ticket can be imported by `creating an issue
<https://docs.gitlab.com/ee/api/issues.html#new-issue>`_ , `commenting on it
<https://docs.gitlab.com/ee/api/notes.html#create-new-issue-note>`_, eventually close it
by `editing the issue
<https://docs.gitlab.com/ee/api/notes.html#create-new-issue-note>`_ and add any
attachments by `uploading file to project
<https://docs.gitlab.com/ee/api/projects.html#upload-a-file>`_.
I tested this on `ARC project ticket <https://pagure.io/fedora-infra/arc/issue/59>`_.
@ -53,11 +54,11 @@ I tested this on `ARC project ticket <https://pagure.io/fedora-infra/arc/issue/5
.. code-block:: json
{
"created_at": "2023-01-19T11:41:40Z",
"title": "Investigate the GitLab API for Pagure to Gitlab importer",
"description": "Investigate the GitLab API for Pagure to Gitlab importer ARC investigation. This ticket will also work as a test ticket in investigation."
}
{
"created_at": "2023-01-19T11:41:40Z",
"title": "Investigate the GitLab API for Pagure to Gitlab importer",
"description": "Investigate the GitLab API for Pagure to Gitlab importer ARC investigation. This ticket will also work as a test ticket in investigation."
}
This creates the `issue on GitLab <https://gitlab.com/testgroup519/arc/-/issues/1>`_.
@ -65,45 +66,44 @@ I tested this on `ARC project ticket <https://pagure.io/fedora-infra/arc/issue/5
.. code-block:: json
{
"created_at": "2023-01-19T12:59:59Z",
"body": "Here's a sample comment as you requested @zlopez."
}
This creates `comment <https://gitlab.com/testgroup519/arc/-/issues/1#note_1245817484>`_ on
the previously created issue. In this case the comment was created by user, who API key is used
to execute API call and according to API documentation it's not possible to change it.
We can overcome this by adding relevant information to each comment or issue. For example
the payload can look like this:
{
"created_at": "2023-01-19T12:59:59Z",
"body": "Here's a sample comment as you requested @zlopez."
}
This creates `comment
<https://gitlab.com/testgroup519/arc/-/issues/1#note_1245817484>`_ on the previously
created issue. In this case the comment was created by user, who API key is used to
execute API call and according to API documentation it's not possible to change it.
We can overcome this by adding relevant information to each comment or issue. For
example the payload can look like this:
.. code-block:: json
{
"created_at": "2023-01-19T12:59:59Z",
"body": "(This comment was added by [t0xic0der](https://accounts.fedoraproject.org/user/t0xic0der/))\n\nHere's a sample comment as you requested @zlopez."
}
{
"created_at": "2023-01-19T12:59:59Z",
"body": "(This comment was added by [t0xic0der](https://accounts.fedoraproject.org/user/t0xic0der/))\n\nHere's a sample comment as you requested @zlopez."
}
The information about GitLab account could be added as well. This could be obtained by using
`users API call <https://docs.gitlab.com/ee/api/users.html>`_ with `mail` as parameter. If the
user is found it will username of the user on GitLab. The final requests could look like this:
The information about GitLab account could be added as well. This could be obtained
by using `users API call <https://docs.gitlab.com/ee/api/users.html>`_ with `mail` as
parameter. If the user is found it will username of the user on GitLab. The final
requests could look like this:
.. code-block:: json
{
"created_at": "2023-01-19T12:59:59Z",
"body": "(This comment was added by @t0xic0der [FAS](https://accounts.fedoraproject.org/user/t0xic0der/))\n\nHere's a sample comment as you requested @zlopez."
}
{
"created_at": "2023-01-19T12:59:59Z",
"body": "(This comment was added by @t0xic0der [FAS](https://accounts.fedoraproject.org/user/t0xic0der/))\n\nHere's a sample comment as you requested @zlopez."
}
Importing pull requests
-----------------------
To import the pull requests it's possible to use `Create MR API call
<https://docs.gitlab.com/ee/api/merge_requests.html#create-mr>`_. However this doesn't allow
us to open pull requests from any repository that is not hosted on GitLab. Following are
the options that could be leveraged by the user for migration:
<https://docs.gitlab.com/ee/api/merge_requests.html#create-mr>`_. However this doesn't
allow us to open pull requests from any repository that is not hosted on GitLab.
Following are the options that could be leveraged by the user for migration:
1. Don't import any pull requests
@ -112,30 +112,31 @@ the options that could be leveraged by the user for migration:
2. Merge all the pull requests before migration
This is the best option for migration. Migration of the merged code is easy and there wouldn't
be any information loss when migrating.
This is the best option for migration. Migration of the merged code is easy and there
wouldn't be any information loss when migrating.
3. Migrate forks to GitLab as well
This will allow us to create pull requests from migrated forks of the original repository.
But this option means migrating plenty of git repositories, just to open the pull requests.
If this option is chosen, I'm recommending to only limit this to open pull requests on Pagure.
This will allow us to create pull requests from migrated forks of the original
repository. But this option means migrating plenty of git repositories, just to open
the pull requests. If this option is chosen, I'm recommending to only limit this to
open pull requests on Pagure.
4. Create branches on migrated repository
It's easy to open pull request from branch on the same repository, but this will mean that
tool will need to go through all the forks, add them as remotes and then add those as
local branches, which allows us to easily create a pull request. I don't recommend using
this approach as it needs plenty of adjustments on the migrated repository before the actual
migration.
It's easy to open pull request from branch on the same repository, but this will mean
that tool will need to go through all the forks, add them as remotes and then add
those as local branches, which allows us to easily create a pull request. I don't
recommend using this approach as it needs plenty of adjustments on the migrated
repository before the actual migration.
In addition to the issue mentioned above the pull requests will have the same issue with author
as import of tickets. The author will be set to whoever generated the API key.
In addition to the issue mentioned above the pull requests will have the same issue with
author as import of tickets. The author will be set to whoever generated the API key.
Conclusion
----------
Using the REST API v4 is not useful for our purpose, because it will set author of every pull
request, issue and comment to user running the migration tool. This could be mitigated by
adding the information directly to the pull request, issue or comment with link to FAS account
of the user.
Using the REST API v4 is not useful for our purpose, because it will set author of every
pull request, issue and comment to user running the migration tool. This could be
mitigated by adding the information directly to the pull request, issue or comment with
link to FAS account of the user.

210
docs/pagure2gitlab/gitlab_file_import.rst
View file

@ -4,141 +4,145 @@ GitLab import from a file
=========================
GitLab provides option to `import a new project from a file
<https://docs.gitlab.com/ee/api/project_import_export.html>`_. This option is created for migrating
projects from one GitLab instance to another. In case of Pagure to GitLab importer we need to adapt
Pagure data to file format used by GitLab. This document will investigate that option.
<https://docs.gitlab.com/ee/api/project_import_export.html>`_. This option is created
for migrating projects from one GitLab instance to another. In case of Pagure to GitLab
importer we need to adapt Pagure data to file format used by GitLab. This document will
investigate that option.
GitLab file format
------------------
For purpose of investigation of the GitLab export format I tried to export `test project
<https://gitlab.com/testgroup519/arc>`_ I created during the investigation of GitLab API.
See :doc:`gitlab`.
<https://gitlab.com/testgroup519/arc>`_ I created during the investigation of GitLab
API. See :doc:`gitlab`.
The export will generate one archive in `tar.gz` format. This archive contains a following
directory structure::
The export will generate one archive in `tar.gz` format. This archive contains a
following directory structure:
2023-01-20_11-48-813_testgroup519_arc_export
├── GITLAB_REVISION
├── GITLAB_VERSION
├── lfs-objects
│ └── 45a5d77993d525cdda15d08e63c34339a1bf49a43756a05908082bb04b4c4087
├── lfs-objects.json
├── project.bundle
├── project.design.bundle
├── snippets
├── tree
│ ├── project
│ │ ├── auto_devops.ndjson
│ │ ├── boards.ndjson
│ │ ├── ci_cd_settings.ndjson
│ │ ├── ci_pipelines.ndjson
│ │ ├── container_expiration_policy.ndjson
│ │ ├── custom_attributes.ndjson
│ │ ├── error_tracking_setting.ndjson
│ │ ├── external_pull_requests.ndjson
│ │ ├── issues.ndjson
│ │ ├── labels.ndjson
│ │ ├── merge_requests.ndjson
│ │ ├── metrics_setting.ndjson
│ │ ├── milestones.ndjson
│ │ ├── pipeline_schedules.ndjson
│ │ ├── project_badges.ndjson
│ │ ├── project_feature.ndjson
│ │ ├── project_members.ndjson
│ │ ├── prometheus_metrics.ndjson
│ │ ├── protected_branches.ndjson
│ │ ├── protected_environments.ndjson
│ │ ├── protected_tags.ndjson
│ │ ├── push_rule.ndjson
│ │ ├── releases.ndjson
│ │ ├── security_setting.ndjson
│ │ ├── service_desk_setting.ndjson
│ │ └── snippets.ndjson
│ └── project.json
├── uploads
│ └── 8b4f7247f154d0b77c5d7d13e16cb904
│ └── Infra___Releng_2022.jpg
└── VERSION
.. code-block::
7 directories, 35 files
2023-01-20_11-48-813_testgroup519_arc_export
├── GITLAB_REVISION
├── GITLAB_VERSION
├── lfs-objects
│ └── 45a5d77993d525cdda15d08e63c34339a1bf49a43756a05908082bb04b4c4087
├── lfs-objects.json
├── project.bundle
├── project.design.bundle
├── snippets
├── tree
│ ├── project
│ │ ├── auto_devops.ndjson
│ │ ├── boards.ndjson
│ │ ├── ci_cd_settings.ndjson
│ │ ├── ci_pipelines.ndjson
│ │ ├── container_expiration_policy.ndjson
│ │ ├── custom_attributes.ndjson
│ │ ├── error_tracking_setting.ndjson
│ │ ├── external_pull_requests.ndjson
│ │ ├── issues.ndjson
│ │ ├── labels.ndjson
│ │ ├── merge_requests.ndjson
│ │ ├── metrics_setting.ndjson
│ │ ├── milestones.ndjson
│ │ ├── pipeline_schedules.ndjson
│ │ ├── project_badges.ndjson
│ │ ├── project_feature.ndjson
│ │ ├── project_members.ndjson
│ │ ├── prometheus_metrics.ndjson
│ │ ├── protected_branches.ndjson
│ │ ├── protected_environments.ndjson
│ │ ├── protected_tags.ndjson
│ │ ├── push_rule.ndjson
│ │ ├── releases.ndjson
│ │ ├── security_setting.ndjson
│ │ ├── service_desk_setting.ndjson
│ │ └── snippets.ndjson
│ └── project.json
├── uploads
│ └── 8b4f7247f154d0b77c5d7d13e16cb904
│ └── Infra___Releng_2022.jpg
└── VERSION
7 directories, 35 files
Following is the explanation of some of the files found in the archive:
* GitLab metadata files (version and revision)
* `.bundle` file which is created by `git bundle <https://git-scm.com/docs/git-bundle>`_ command.
You can easily look at the content of `.bundle` file by using `git clone` command.
* `.design.bundle` contains all the attachments from issues and merge requests. It is
a repository file bundled by `git bundle <https://git-scm.com/docs/git-bundle>`_ command.
* `lsf-object.json` contains list of hashes of designs and their mapping to issue id or merge
request id. This is something we can skip, because Pagure doesn't have this feature.
* `VERSION` file contains version, but I was not able what this version refers to. My assumption is
that it's version of the export tool.
* `lfs-objects/` folder contains all the designs named by hash. This is something we can skip,
because Pagure doesn't have this feature.
* `snippets/` folder contains `GitLab snippets <https://docs.gitlab.com/ee/user/snippets.html>`_.
* `tree/project.json` file contains all the project metadata in JSON format.
* `tree/project/` contains files in `ndjson format <http://ndjson.org/>`_ describing various
objects defined in GitLab project. For purpose of this investigation only `issues.ndjson` and
`merge_requests.ndjson` are important for us.
* `uploads/` folder contains all the attachments from issues or merge requests.
- GitLab metadata files (version and revision)
- `.bundle` file which is created by `git bundle <https://git-scm.com/docs/git-bundle>`_
command. You can easily look at the content of `.bundle` file by using `git clone`
command.
- `.design.bundle` contains all the attachments from issues and merge requests. It is a
repository file bundled by `git bundle <https://git-scm.com/docs/git-bundle>`_
command.
- `lsf-object.json` contains list of hashes of designs and their mapping to issue id or
merge request id. This is something we can skip, because Pagure doesn't have this
feature.
- `VERSION` file contains version, but I was not able what this version refers to. My
assumption is that it's version of the export tool.
- `lfs-objects/` folder contains all the designs named by hash. This is something we can
skip, because Pagure doesn't have this feature.
- `snippets/` folder contains `GitLab snippets
<https://docs.gitlab.com/ee/user/snippets.html>`_.
- `tree/project.json` file contains all the project metadata in JSON format.
- `tree/project/` contains files in `ndjson format <http://ndjson.org/>`_ describing
various objects defined in GitLab project. For purpose of this investigation only
`issues.ndjson` and `merge_requests.ndjson` are important for us.
- `uploads/` folder contains all the attachments from issues or merge requests.
Conversion of Pagure project to GitLab file formats
---------------------------------------------------
For purpose of the investigation I tried to convert `ARC project <https://pagure.io/fedora-infra/arc>`_
hosted on Pagure to GitLab import format. For this purpose I started with the export generated by GitLab
and changed files to correspond to what I want to import.
For purpose of the investigation I tried to convert `ARC project
<https://pagure.io/fedora-infra/arc>`_ hosted on Pagure to GitLab import format. For
this purpose I started with the export generated by GitLab and changed files to
correspond to what I want to import.
Here is the list of all files that I needed to prepare and their content with explanation:
Here is the list of all files that I needed to prepare and their content with
explanation:
* `project.bundle` is a binary bundle file created by `git bundle <https://git-scm.com/docs/git-bundle>`_
command. It was created by running `git bundle create project.bundle --all` inside ARC project repository.
* `tree/project/issues.ndjson` contains issues description in `ndjson format <http://ndjson.org/>`_.
The file contains `project_id` or `author_id` set to 0, instead it contains `author` object with
FAS username and public FAS e-mail. Unfortunately if the `author_id` isn't recognized by GitLab it
will create the issue or comment as a user who is providing the import, completely ignoring the author
- `project.bundle` is a binary bundle file created by `git bundle
<https://git-scm.com/docs/git-bundle>`_ command. It was created by running `git bundle
create project.bundle --all` inside ARC project repository.
- `tree/project/issues.ndjson` contains issues description in `ndjson format
<http://ndjson.org/>`_. The file contains `project_id` or `author_id` set to 0,
instead it contains `author` object with FAS username and public FAS e-mail.
Unfortunately if the `author_id` isn't recognized by GitLab it will create the issue
or comment as a user who is providing the import, completely ignoring the author
object in JSON.
.. code-block:: json
{"title":"Investigate the GitLab API for Pagure to Gitlab importer","author_id":0,"author":{"username": "zlopez","email": "michal.konecny@pacse.eu"},"project_id":42729361,"created_at":"2023-01-19T11:41:40.000Z","updated_at":"2023-01-19T14:06:47.659Z","description":"Investigate the GitLab API for Pagure to Gitlab importer ARC investigation. This ticket will also work as a test ticket in investigation.","iid":1,"updated_by_id":null,"weight":null,"confidential":false,"due_date":null,"lock_version":0,"time_estimate":0,"relative_position":513,"last_edited_at":null,"last_edited_by_id":null,"discussion_locked":null,"closed_at":"2023-01-19T14:06:47.641Z","closed_by_id":3072529,"health_status":null,"external_key":null,"issue_type":"issue","state":"closed","events":[{"project_id":42729361,"author_id":3072529,"created_at":"2023-01-19T13:07:11.164Z","updated_at":"2023-01-19T13:07:11.164Z","action":"created","target_type":"Issue","fingerprint":null},{"project_id":42729361,"author_id":3072529,"created_at":"2023-01-19T14:06:47.712Z","updated_at":"2023-01-19T14:06:47.712Z","action":"closed","target_type":"Issue","fingerprint":null}],"timelogs":[],"notes":[{"note":"Here's a sample comment as you requested @zlopez.","noteable_type":"Issue","author_id":3072529,"created_at":"2023-01-19T12:59:59.000Z","updated_at":"2023-01-19T12:59:59.000Z","project_id":42729361,"attachment":{"url":null},"line_code":null,"commit_id":null,"st_diff":null,"system":false,"updated_by_id":null,"type":null,"position":null,"original_position":null,"resolved_at":null,"resolved_by_id":null,"discussion_id":"f98cdeabaaec68ae453e1dbf5d9e535fbbcede0a","change_position":null,"resolved_by_push":null,"confidential":null,"last_edited_at":"2023-01-19T12:59:59.000Z","author":{"name":"Zlopez"},"award_emoji":[],"events":[{"project_id":42729361,"author_id":3072529,"created_at":"2023-01-19T13:13:21.071Z","updated_at":"2023-01-19T13:13:21.071Z","action":"commented","target_type":"Note","fingerprint":null}]}],"label_links":[],"resource_label_events":[],"resource_milestone_events":[],"resource_state_events":[{"user_id":3072529,"created_at":"2023-01-19T14:06:47.734Z","state":"closed","source_commit":null,"close_after_error_tracking_resolve":false,"close_auto_resolve_prometheus_alert":false}],"designs":[],"design_versions":[],"issue_assignees":[],"zoom_meetings":[],"award_emoji":[],"resource_iteration_events":[]}
{"title":"Test open issue","author_id":0,"author":{"username": "akashdeep","email": "akashdeep.dhar@gmail.com"},"project_id":42729361,"created_at":"2023-01-19T14:07:05.823Z","updated_at":"2023-01-20T11:48:02.495Z","description":"Test open issue","iid":2,"updated_by_id":null,"weight":null,"confidential":false,"due_date":null,"lock_version":0,"time_estimate":0,"relative_position":1026,"last_edited_at":null,"last_edited_by_id":null,"discussion_locked":null,"closed_at":null,"closed_by_id":null,"health_status":null,"external_key":null,"issue_type":"issue","state":"opened","events":[{"project_id":42729361,"author_id":3072529,"created_at":"2023-01-19T14:07:05.930Z","updated_at":"2023-01-19T14:07:05.930Z","action":"created","target_type":"Issue","fingerprint":null}],"timelogs":[],"notes":[{"note":"![Infra___Releng_2022](/uploads/8b4f7247f154d0b77c5d7d13e16cb904/Infra___Releng_2022.jpg)","noteable_type":"Issue","author_id":3072529,"created_at":"2023-01-20T11:48:02.435Z","updated_at":"2023-01-20T11:48:02.435Z","project_id":42729361,"attachment":{"url":null},"line_code":null,"commit_id":null,"st_diff":null,"system":false,"updated_by_id":null,"type":null,"position":null,"original_position":null,"resolved_at":null,"resolved_by_id":null,"discussion_id":"30302c7dee98663fcfca845a2ec2715eb3e35e4f","change_position":null,"resolved_by_push":null,"confidential":null,"last_edited_at":"2023-01-20T11:48:02.435Z","author":{"name":"Zlopez"},"award_emoji":[],"events":[{"project_id":42729361,"author_id":3072529,"created_at":"2023-01-20T11:48:02.617Z","updated_at":"2023-01-20T11:48:02.617Z","action":"commented","target_type":"Note","fingerprint":null}]},{"note":"added [1 design](/testgroup519/arc/-/issues/2/designs?version=490993)","noteable_type":"Issue","author_id":3072529,"created_at":"2023-01-19T14:07:45.310Z","updated_at":"2023-01-19T14:07:45.315Z","project_id":42729361,"attachment":{"url":null},"line_code":null,"commit_id":null,"st_diff":null,"system":true,"updated_by_id":null,"type":null,"position":null,"original_position":null,"resolved_at":null,"resolved_by_id":null,"discussion_id":"e15e7c584cc7e6c7e298529f034f0b55eeacca90","change_position":null,"resolved_by_push":null,"confidential":null,"last_edited_at":"2023-01-19T14:07:45.315Z","author":{"name":"Zlopez"},"award_emoji":[],"system_note_metadata":{"commit_count":null,"action":"designs_added","created_at":"2023-01-19T14:07:45.343Z","updated_at":"2023-01-19T14:07:45.343Z"},"events":[]}],"label_links":[],"resource_label_events":[],"resource_milestone_events":[],"resource_state_events":[],"designs":[{"project_id":42729361,"filename":"Infra___Releng_2022.jpg","relative_position":null,"iid":1,"notes":[]}],"design_versions":[{"sha":"69419c215f53d401c1b3c451e6fc08e3351d2679","created_at":"2023-01-19T14:07:45.233Z","author_id":3072529,"actions":[{"event":"creation","design":{"project_id":42729361,"filename":"Infra___Releng_2022.jpg","relative_position":null,"iid":1}}]}],"issue_assignees":[],"zoom_meetings":[],"award_emoji":[],"resource_iteration_events":[]}
{"title":"Investigate the GitLab API for Pagure to Gitlab importer","author_id":0,"author":{"username": "zlopez","email": "michal.konecny@pacse.eu"},"project_id":42729361,"created_at":"2023-01-19T11:41:40.000Z","updated_at":"2023-01-19T14:06:47.659Z","description":"Investigate the GitLab API for Pagure to Gitlab importer ARC investigation. This ticket will also work as a test ticket in investigation.","iid":1,"updated_by_id":null,"weight":null,"confidential":false,"due_date":null,"lock_version":0,"time_estimate":0,"relative_position":513,"last_edited_at":null,"last_edited_by_id":null,"discussion_locked":null,"closed_at":"2023-01-19T14:06:47.641Z","closed_by_id":3072529,"health_status":null,"external_key":null,"issue_type":"issue","state":"closed","events":[{"project_id":42729361,"author_id":3072529,"created_at":"2023-01-19T13:07:11.164Z","updated_at":"2023-01-19T13:07:11.164Z","action":"created","target_type":"Issue","fingerprint":null},{"project_id":42729361,"author_id":3072529,"created_at":"2023-01-19T14:06:47.712Z","updated_at":"2023-01-19T14:06:47.712Z","action":"closed","target_type":"Issue","fingerprint":null}],"timelogs":[],"notes":[{"note":"Here's a sample comment as you requested @zlopez.","noteable_type":"Issue","author_id":3072529,"created_at":"2023-01-19T12:59:59.000Z","updated_at":"2023-01-19T12:59:59.000Z","project_id":42729361,"attachment":{"url":null},"line_code":null,"commit_id":null,"st_diff":null,"system":false,"updated_by_id":null,"type":null,"position":null,"original_position":null,"resolved_at":null,"resolved_by_id":null,"discussion_id":"f98cdeabaaec68ae453e1dbf5d9e535fbbcede0a","change_position":null,"resolved_by_push":null,"confidential":null,"last_edited_at":"2023-01-19T12:59:59.000Z","author":{"name":"Zlopez"},"award_emoji":[],"events":[{"project_id":42729361,"author_id":3072529,"created_at":"2023-01-19T13:13:21.071Z","updated_at":"2023-01-19T13:13:21.071Z","action":"commented","target_type":"Note","fingerprint":null}]}],"label_links":[],"resource_label_events":[],"resource_milestone_events":[],"resource_state_events":[{"user_id":3072529,"created_at":"2023-01-19T14:06:47.734Z","state":"closed","source_commit":null,"close_after_error_tracking_resolve":false,"close_auto_resolve_prometheus_alert":false}],"designs":[],"design_versions":[],"issue_assignees":[],"zoom_meetings":[],"award_emoji":[],"resource_iteration_events":[]}
{"title":"Test open issue","author_id":0,"author":{"username": "akashdeep","email": "akashdeep.dhar@gmail.com"},"project_id":42729361,"created_at":"2023-01-19T14:07:05.823Z","updated_at":"2023-01-20T11:48:02.495Z","description":"Test open issue","iid":2,"updated_by_id":null,"weight":null,"confidential":false,"due_date":null,"lock_version":0,"time_estimate":0,"relative_position":1026,"last_edited_at":null,"last_edited_by_id":null,"discussion_locked":null,"closed_at":null,"closed_by_id":null,"health_status":null,"external_key":null,"issue_type":"issue","state":"opened","events":[{"project_id":42729361,"author_id":3072529,"created_at":"2023-01-19T14:07:05.930Z","updated_at":"2023-01-19T14:07:05.930Z","action":"created","target_type":"Issue","fingerprint":null}],"timelogs":[],"notes":[{"note":"![Infra___Releng_2022](/uploads/8b4f7247f154d0b77c5d7d13e16cb904/Infra___Releng_2022.jpg)","noteable_type":"Issue","author_id":3072529,"created_at":"2023-01-20T11:48:02.435Z","updated_at":"2023-01-20T11:48:02.435Z","project_id":42729361,"attachment":{"url":null},"line_code":null,"commit_id":null,"st_diff":null,"system":false,"updated_by_id":null,"type":null,"position":null,"original_position":null,"resolved_at":null,"resolved_by_id":null,"discussion_id":"30302c7dee98663fcfca845a2ec2715eb3e35e4f","change_position":null,"resolved_by_push":null,"confidential":null,"last_edited_at":"2023-01-20T11:48:02.435Z","author":{"name":"Zlopez"},"award_emoji":[],"events":[{"project_id":42729361,"author_id":3072529,"created_at":"2023-01-20T11:48:02.617Z","updated_at":"2023-01-20T11:48:02.617Z","action":"commented","target_type":"Note","fingerprint":null}]},{"note":"added [1 design](/testgroup519/arc/-/issues/2/designs?version=490993)","noteable_type":"Issue","author_id":3072529,"created_at":"2023-01-19T14:07:45.310Z","updated_at":"2023-01-19T14:07:45.315Z","project_id":42729361,"attachment":{"url":null},"line_code":null,"commit_id":null,"st_diff":null,"system":true,"updated_by_id":null,"type":null,"position":null,"original_position":null,"resolved_at":null,"resolved_by_id":null,"discussion_id":"e15e7c584cc7e6c7e298529f034f0b55eeacca90","change_position":null,"resolved_by_push":null,"confidential":null,"last_edited_at":"2023-01-19T14:07:45.315Z","author":{"name":"Zlopez"},"award_emoji":[],"system_note_metadata":{"commit_count":null,"action":"designs_added","created_at":"2023-01-19T14:07:45.343Z","updated_at":"2023-01-19T14:07:45.343Z"},"events":[]}],"label_links":[],"resource_label_events":[],"resource_milestone_events":[],"resource_state_events":[],"designs":[{"project_id":42729361,"filename":"Infra___Releng_2022.jpg","relative_position":null,"iid":1,"notes":[]}],"design_versions":[{"sha":"69419c215f53d401c1b3c451e6fc08e3351d2679","created_at":"2023-01-19T14:07:45.233Z","author_id":3072529,"actions":[{"event":"creation","design":{"project_id":42729361,"filename":"Infra___Releng_2022.jpg","relative_position":null,"iid":1}}]}],"issue_assignees":[],"zoom_meetings":[],"award_emoji":[],"resource_iteration_events":[]}
Importing the archive to GitLab
-------------------------------
Archive for the migration is prepared by executing `tar -czvf test_arc_export.tar.gz .` command. This needs
to be executed in the root folder of the prepared file structure, otherwise the import will fail with
`No such file or directory`.
Archive for the migration is prepared by executing `tar -czvf test_arc_export.tar.gz .`
command. This needs to be executed in the root folder of the prepared file structure,
otherwise the import will fail with `No such file or directory`.
To import the archive to GitLab `API call <https://docs.gitlab.com/ee/api/project_import_export.html#import-a-file>`_
could be used. Here is the full API call made by `curl`::
To import the archive to GitLab `API call
<https://docs.gitlab.com/ee/api/project_import_export.html#import-a-file>`_ could be
used. Here is the full API call made by `curl`:
curl --request POST --header "PRIVATE-TOKEN: XXX" --form "namespace=testgroup519" --form "path=arc2" --form "file=@test_arc_export.tar.gz" "https://gitlab.com/api/v4/projects/import"
.. code-block::
curl --request POST --header "PRIVATE-TOKEN: XXX" --form "namespace=testgroup519" --form "path=arc2" --form "file=@test_arc_export.tar.gz" "https://gitlab.com/api/v4/projects/import"
To check for any error in the import use GitLab `import status API call
<https://docs.gitlab.com/ee/api/project_import_export.html#import-status>`_. This could be made by `curl`::
<https://docs.gitlab.com/ee/api/project_import_export.html#import-status>`_. This could
be made by `curl`:
curl --header "PRIVATE-TOKEN: XXX" "https://gitlab.com/api/v4/projects/<id_returned_by_import_call>/import"
.. code-block::
curl --header "PRIVATE-TOKEN: XXX" "https://gitlab.com/api/v4/projects/<id_returned_by_import_call>/import"
Conclusion
----------
At this point I ended up with the investigation, because the situation is the same as in case of using API.
Which is much more convenient to use and provides a better response in case of errors (I spent two days trying
to debug `No such file or directory [FILTERED]` error message).
At this point I ended up with the investigation, because the situation is the same as in
case of using API. Which is much more convenient to use and provides a better response
in case of errors (I spent two days trying to debug `No such file or directory
[FILTERED]` error message).

74
docs/pagure2gitlab/index.rst
View file

@ -4,85 +4,85 @@ Pagure to GitLab Importer
Purpose
-------
With Fedora and CentOS now having an official namespace on GitLab multiple projects wants to
migrate their repositories from `Pagure <https://pagure.io/>`_ to `GitLab <https://gitlab.com>`_.
This initiative is aimed to provide an easy way to migrate those projects.
With Fedora and CentOS now having an official namespace on GitLab multiple projects
wants to migrate their repositories from `Pagure <https://pagure.io/>`_ to `GitLab
<https://gitlab.com>`_. This initiative is aimed to provide an easy way to migrate those
projects.
Resources
---------
* Initiative proposal: https://pagure.io/cpe/initiatives-proposal/issue/25
* Pagure documentation: https://pagure.io/docs/pagure/
* GitLab documentation: https://docs.gitlab.com/
- Initiative proposal: https://pagure.io/cpe/initiatives-proposal/issue/25
- Pagure documentation: https://pagure.io/docs/pagure/
- GitLab documentation: https://docs.gitlab.com/
Requirements
------------
* Self-service tool
* Git commit history preservation
* Ability to migrate pagure issues
* Command line interface
* Ability to migrate files uploaded to PRs and issues
* Ability to migrate pull-requests
* Mapping of Fedora identity to GitLab users
* Tool should be usable by both CentOS and Fedora users
* Ability to access the correct namespace on GitLab
- Self-service tool
- Git commit history preservation
- Ability to migrate pagure issues
- Command line interface
- Ability to migrate files uploaded to PRs and issues
- Ability to migrate pull-requests
- Mapping of Fedora identity to GitLab users
- Tool should be usable by both CentOS and Fedora users
- Ability to access the correct namespace on GitLab
Nice to have
------------
List of features that would be nice to have.
* Graphical UI
* Web interface
* Syncing the FAS usernames to GitLab as users migrate for already migrated projects
- Graphical UI
- Web interface
- Syncing the FAS usernames to GitLab as users migrate for already migrated projects
Investigation
-------------
Following are the investigations of Pagure options to export and GitLab options to import.
Following are the investigations of Pagure options to export and GitLab options to
import.
.. toctree::
:maxdepth: 1
pagure
pagureinfo
gitlab
gitlab_file_import
protop2g
Conclusions
-----------
Creating the tool for migration is possible, but with a few caveats.
We don't recommend to migrate pull requests
(:ref:`pagure2gitlab/gitlab:Importing pull requests`).
We don't recommend to migrate pull requests (:ref:`pagure2gitlab/gitlab:Importing pull
requests`).
We didn't investigate the possibility of providing web interface, it's definitely possible,
but it will add much to complexity of the tool and we need to host it and deploy as well.
We didn't investigate the possibility of providing web interface, it's definitely
possible, but it will add much to complexity of the tool and we need to host it and
deploy as well.
The main issue with the migration is that all the comments, issues are created by the user
that owns the API token, which is made to execute the API calls. This issue is explained
in :ref:`pagure2gitlab/gitlab:Importing ticket` with possible proposed solution.
The user that is using the tool will need to have permissions to create new repositories in
the GitLab group, to which the repository will be migrated.
The main issue with the migration is that all the comments, issues are created by the
user that owns the API token, which is made to execute the API calls. This issue is
explained in :ref:`pagure2gitlab/gitlab:Importing ticket` with possible proposed
solution.
The user that is using the tool will need to have permissions to create new repositories
in the GitLab group, to which the repository will be migrated.
Proposed Roadmap
----------------
1. Create the tool for migration using GitLab API
2. (Optional) Create a bot user for migration and share the API key with trusted people. This
will help to migrate some of the projects as neutral user instead of the user, who is using
the tool.
2. (Optional) Create a bot user for migration and share the API key with trusted people.
This will help to migrate some of the projects as neutral user instead of the user,
who is using the tool.
3. Prepare the documentation for migration process.
4. Announce the migration tool is available.
Estimate of work
----------------
This work will need 2 developers.
The estimation for this project is 4 weeks.
This work will need 2 developers. The estimation for this project is 4 weeks.

122
docs/pagure2gitlab/pagure.rst
View file

@ -1,122 +0,0 @@
.. _pagure:
Exporting Namespace Assets From Pagure
====
A lot of projects related to Fedora Project and CentOS Project reside in Pagure
which makes it important for us to understand the ways they can be interacted
with and moved around when the need arises.
Elements for interaction
----
In this context, we are looking into interacting with the following elements -
Issue tickets
^^^^
These can either be requests for adding new features, reporting of a bug,
proposal of an idea or anything else. These need to be available in the place
where the namespace assets get transferred to, in order to retain the context
around the probable action items.
Repository assets
^^^^
These are the actual files stored in the Git repository that pertains to the
actual project. This can either be a codebase, a collection of documentation,
a group of artworks or anything else. This happens to be the primary subject
for the transfer.
We are avoiding interacting with the following elements -
Pull requests
^^^^
Not only does it require for the pull requests to be artificially created by a
stub user (say, a bot user of some kind as we cannot guarantee the availability
of the GitLab user pertaining to the related Pagure user) in the destination
namespace but it also needs the source of the pull request (which can either be
a fork of the source namespace of a branch inside the same repository) to be
available, which might not always be the case. We strongly recommend merging
all pending pull requests that are needed and closing those that are not,
before the transfer process begins.
Ways of interaction
----
The elements that we would want to interact with, can be done so using the
following ways.
Issue tickets
^^^^
* *Pagure API*
This provides an `extensive method <https://pagure.io/api/0/>`_ to `interact
with the issue tickets <https://pagure.io/api/0/#issues-tab>`_ pertaining to
a certain source namespace. This can be expanded upon to receive information
like issue title, issue content, assignees, tags, creation date, modified
date, comments etc. Depending on what information and how much amount of it
is requested from the API server, the interactions can be done quickly too,
paving the way for automation. The service does not seem to have rate
limiting too and hence, can be used without worry for interacting with bigger
repositories.
* *The good*
* A well structured API query with limited scope can ensure that multiple
requests are made and addressed quickly.
* This does not require any local storage space to be available in order to
make requests, unless caching is employed.
* *The bad*
* Extra assets bound to the issue ticket like images, documentation, media
files etc. cannot be reliably accessed.
* Service can be slow at times, leading to timeouts and hence, a condition
to address those during automation is needed.
* *Issues Git URL*
For the users that have a commit access to a certain source namespace, they
should be able to see an option for listing "Other Git URLs" on the project
page. Although Pagure provides methods to clone both "Issues" and "Pull
Requests" using those special Git URLs, we would only talk about that for the
former. For instance, the folks having a commit access to the
`fedora-websites <https://pagure.io/fedora-websites>`_ repository should be
able to access the issue tickets using the URL
`ssh://git@pagure.io/tickets/fedora-websites.git <ssh://git@pagure.io/tickets/fedora-websites.git>`_.
* *The good*
* A clone of the issues repository can help receive all the issue tickets
ever created in one fell swoop.
* Extra assets bound to the issue ticket like images, documentation, media
files etc. can be reliably accessed.
* *The bad*
* It can take a long time to fetch the issues repository in its entirety if
the repository has had too many of those.
* This mandatorily requires the use of a local storage space to be able to
clone and make changes locally before publish.
Repository assets
^^^^
* *Default Git URL*
Folks can access the repository assets of the source namespace using the
default Git URL. Depending on the availability of the API keys for the users,
they can choose to either interact with the repository assets with the use of
either the HTTPS URL (for example, it would be
`https://pagure.io/fedora-websites.git <https://pagure.io/fedora-websites.git>`_
for the fedora-websites source namespace) or the SSH URL (for example, it
would be
`ssh://git@pagure.io/fedora-websites.git <ssh://git@pagure.io/fedora-websites.git>`_
for the fedora-websites source namespace). We would not go in detail with the
advantages and disadvantages of this method as there seems to be only one way
to interact with repository assets.

107
docs/pagure2gitlab/pagureinfo.rst Normal file
View file

@ -0,0 +1,107 @@
.. _pagureinfo:
Exporting Namespace Assets From Pagure
======================================
A lot of projects related to Fedora Project and CentOS Project reside in Pagure which
makes it important for us to understand the ways they can be interacted with and moved
around when the need arises.
Elements for interaction
------------------------
In this context, we are looking into interacting with the following elements:
**Issue tickets**
These can either be requests for adding new features, reporting of a bug, proposal
of an idea or anything else. These need to be available in the place where the
namespace assets get transferred to, in order to retain the context around the
probable action items.
**Repository assets**
These are the actual files stored in the Git repository that pertains to the actual
project. This can either be a codebase, a collection of documentation, a group of
artworks or anything else. This happens to be the primary subject for the transfer.
We are avoiding interacting with the following elements
**Pull requests**
Not only does it require for the pull requests to be artificially created by a stub
user (say, a bot user of some kind as we cannot guarantee the availability of the
GitLab user pertaining to the related Pagure user) in the destination namespace but
it also needs the source of the pull request (which can either be a fork of the
source namespace of a branch inside the same repository) to be available, which
might not always be the case. We strongly recommend merging all pending pull
requests that are needed and closing those that are not, before the transfer process
begins.
Ways of interaction
-------------------
The elements that we would want to interact with, can be done so using the following
ways.
**Issue tickets**
- *Pagure API*
This provides an `extensive method <https://pagure.io/api/0/>`_ to `interact with
the issue tickets <https://pagure.io/api/0/#issues-tab>`_ pertaining to a certain
source namespace. This can be expanded upon to receive information like issue
title, issue content, assignees, tags, creation date, modified date, comments etc.
Depending on what information and how much amount of it is requested from the API
server, the interactions can be done quickly too, paving the way for automation.
The service does not seem to have rate limiting too and hence, can be used without
worry for interacting with bigger repositories.
- *The good*
- A well structured API query with limited scope can ensure that multiple
requests are made and addressed quickly.
- This does not require any local storage space to be available in order to make
requests, unless caching is employed.
- *The bad*
- Extra assets bound to the issue ticket like images, documentation, media files
etc. cannot be reliably accessed.
- Service can be slow at times, leading to timeouts and hence, a condition to
address those during automation is needed.
- *Issues Git URL*
For the users that have a commit access to a certain source namespace, they should
be able to see an option for listing "Other Git URLs" on the project page.
Although Pagure provides methods to clone both "Issues" and "Pull Requests" using
those special Git URLs, we would only talk about that for the former. For
instance, the folks having a commit access to the `fedora-websites
<https://pagure.io/fedora-websites>`_ repository should be able to access the
issue tickets using the URL ssh://git@pagure.io/tickets/fedora-websites.git.
- *The good*
- A clone of the issues repository can help receive all the issue tickets ever
created in one fell swoop.
- Extra assets bound to the issue ticket like images, documentation, media files
etc. can be reliably accessed.
- *The bad*
- It can take a long time to fetch the issues repository in its entirety if the
repository has had too many of those.
- This mandatorily requires the use of a local storage space to be able to clone
and make changes locally before publish.
Repository assets
~~~~~~~~~~~~~~~~~
- *Default Git URL*
Folks can access the repository assets of the source namespace using the default Git
URL. Depending on the availability of the API keys for the users, they can choose to
either interact with the repository assets with the use of either the HTTPS URL (for
example, it would be https://pagure.io/fedora-websites.git for the fedora-websites
source namespace) or the SSH URL (for example, it would be
ssh://git@pagure.io/fedora-websites.git for the fedora-websites source namespace). We
would not go in detail with the advantages and disadvantages of this method as there
seems to be only one way to interact with repository assets.

273
docs/pagure2gitlab/protop2g.rst
View file

@ -1,183 +1,178 @@
.. _protop2g:
Prototypes built
====
================
In order to be able to better understand what both the Pagure and GitLab APIs
are capable of and how they would be able to be of assistance in helping us
move assets from the source namespace to the destination namespace, we created
a prototype tool called "protop2g".
In order to be able to better understand what both the Pagure and GitLab APIs are
capable of and how they would be able to be of assistance in helping us move assets from
the source namespace to the destination namespace, we created a prototype tool called
"protop2g".
First rendition
----
---------------
The initial version of protop2g was hacked together in around a couple of days
using libraries like
`Requests <https://requests.readthedocs.io/>`_,
`Asciimatics <https://asciimatics.readthedocs.io/en/stable/intro.html>` etc. to
build an interactive text-based user interface.
The initial version of protop2g was hacked together in around a couple of days using
libraries like `Requests <https://requests.readthedocs.io/>`_, `Asciimatics
<https://asciimatics.readthedocs.io/en/stable/intro.html>` etc. to build an interactive
text-based user interface.
The following is a screenshot of the TUI screen where the users are given the
option to enter their source namespace location, API key for accessing the
source namespace location, destination namespace location, API key for
accessing the destination namespace location.
The following is a screenshot of the TUI screen where the users are given the option to
enter their source namespace location, API key for accessing the source namespace
location, destination namespace location, API key for accessing the destination
namespace location.
.. image:: ../_static/tui-protop2g-main.png
:target: ../_static/tui-protop2g-main.png
The following is a screenshot of the TUI screen where the users are shown the
metadata of both the source namespace and destination namespace. They are also
given the option to pick which assets they wish to interact with and transfer,
like repository data, issue tickets etc.
The following is a screenshot of the TUI screen where the users are shown the metadata
of both the source namespace and destination namespace. They are also given the option
to pick which assets they wish to interact with and transfer, like repository data,
issue tickets etc.
.. image:: ../_static/tui-protop2g-repo.png
:target: ../_static/tui-protop2g-repo.png
Unfortunately, this approach was scrapped due to the following reasons :-
* The TUI application approach would not allow for a simpler automation to help
folks who either own and/or maintain a lot of repositories.
* The logic for building the screens and sharing information around the screens
could possibly deviate from the actual feasibility checks.
* The colour palette and the elements used in the TUI might not be accessible
on a variety of terminal and console setups, that people have.
- The TUI application approach would not allow for a simpler automation to help folks
who either own and/or maintain a lot of repositories.
- The logic for building the screens and sharing information around the screens could
possibly deviate from the actual feasibility checks.
- The colour palette and the elements used in the TUI might not be accessible on a
variety of terminal and console setups, that people have.
Second rendition
----
----------------
After the initial version was archived in a different branch, the next version
of protop2g was created using libraries like
`Requests <https://requests.readthedocs.io/>`_,
`Click <https://click.palletsprojects.com/en/8.1.x/>`_,
`GitPython <https://gitpython.readthedocs.io/en/stable/tutorial.html>`_ etc. to
better facilitate an interactive command-line interface.
After the initial version was archived in a different branch, the next version of
protop2g was created using libraries like `Requests
<https://requests.readthedocs.io/>`_, `Click
<https://click.palletsprojects.com/en/8.1.x/>`_, `GitPython
<https://gitpython.readthedocs.io/en/stable/tutorial.html>`_ etc. to better facilitate
an interactive command-line interface.
The following is a console snippet of the help documentation of the overall
usage of the protop2g application.
The following is a console snippet of the help documentation of the overall usage of the
protop2g application.
.. code-block:: shell
.. code-block:: shell
(venv) [t0xic0der@fedorable protop2g]$ protop2g -g xxxxxxxxxxxxxxxxxx -p xxxxxxxxxxxxxxxxxx -s mdapi -d 42823949 -f t0xic0der -t t0xic0der repo --help
Usage: protop2g [OPTIONS] COMMAND [ARGS]...
(venv) [t0xic0der@fedorable protop2g]$ protop2g -g xxxxxxxxxxxxxxxxxx -p xxxxxxxxxxxxxxxxxx -s mdapi -d 42823949 -f t0xic0der -t t0xic0der repo --help
Usage: protop2g [OPTIONS] COMMAND [ARGS]...
Options:
-s, --srce TEXT Source namespace for importing assets from [required]
-d, --dest TEXT Destination namespace for exporting assets to [required]
-p, --pkey TEXT Pagure API key for accessing the source namespace
[required]
-g, --gkey TEXT GitLab API key for accessing the destination namespace
[required]
-f, --fusr TEXT Username of the account that owns the Pagure API key
[required]
-t, --tusr TEXT Username of the account that owns the GitLab API key
[required]
--version Show the version and exit.
--help Show this message and exit.
Options:
-s, --srce TEXT Source namespace for importing assets from [required]
-d, --dest TEXT Destination namespace for exporting assets to [required]
-p, --pkey TEXT Pagure API key for accessing the source namespace
[required]
-g, --gkey TEXT GitLab API key for accessing the destination namespace
[required]
-f, --fusr TEXT Username of the account that owns the Pagure API key
[required]
-t, --tusr TEXT Username of the account that owns the GitLab API key
[required]
--version Show the version and exit.
--help Show this message and exit.
Commands:
repo Initialize transfer of repository assets
tkts Initiate transfer of issue tickets
Commands:
repo Initialize transfer of repository assets
tkts Initiate transfer of issue tickets
The following is a console snippet of the help documentation for the "repo" command of
the protop2g application.
The following is a console snippet of the help documentation for the "repo"
command of the protop2g application.
The options for the command allow for people to be able to choose the branches that
should be transferred.
The options for the command allow for people to be able to choose the branches
that should be transferred.
.. code-block:: shell
.. code-block:: shell
(venv) [t0xic0der@fedorable protop2g]$ protop2g -g xxxxxxxxxxxxxxxxxx -p xxxxxxxxxxxxxxxxxx -s mdapi -d 42823949 -f t0xic0der -t t0xic0der repo --help
Usage: protop2g repo [OPTIONS]
(venv) [t0xic0der@fedorable protop2g]$ protop2g -g xxxxxxxxxxxxxxxxxx -p xxxxxxxxxxxxxxxxxx -s mdapi -d 42823949 -f t0xic0der -t t0xic0der repo --help
Usage: protop2g repo [OPTIONS]
Initialize transfer of repository assets
Initialize transfer of repository assets
Options:
-b, --brcs TEXT List of branches to extract
--help Show this message and exit.
Options:
-b, --brcs TEXT List of branches to extract
--help Show this message and exit.
The following is a console snippet of the help documentation for the "tkts" command of
the protop2g application.
The options for the command allow for people to be able to choose the nature of issues
they want to transfer.
The following is a console snippet of the help documentation for the "tkts"
command of the protop2g application.
.. code-block:: shell
The options for the command allow for people to be able to choose the nature of
issues they want to transfer.
(venv) [t0xic0der@fedorable protop2g]$ protop2g -g xxxxxxxxxxxxxxxxxx -p xxxxxxxxxxxxxxxxxx -s mdapi -d 42823949 -f t0xic0der -t t0xic0der tkts --help
Usage: protop2g tkts [OPTIONS]
.. code-block:: shell
Initiate transfer of issue tickets
(venv) [t0xic0der@fedorable protop2g]$ protop2g -g xxxxxxxxxxxxxxxxxx -p xxxxxxxxxxxxxxxxxx -s mdapi -d 42823949 -f t0xic0der -t t0xic0der tkts --help
Usage: protop2g tkts [OPTIONS]
Options:
-o, --open Extract only the open issue tickets
-c, --shut Extract only the closed issue tickets
-a, --full Extract all the issue tickets
--help Show this message and exit.
Initiate transfer of issue tickets
The following is a console snippet of the protop2g application at work, where it is
transferring the repository assets of all the branches from the source namespace to the
destination namespace.
Options:
-o, --open Extract only the open issue tickets
-c, --shut Extract only the closed issue tickets
-a, --full Extract all the issue tickets
--help Show this message and exit.
The following is a console snippet of the protop2g application at work, where
it is transferring the repository assets of all the branches from the source
namespace to the destination namespace.
.. code-block:: shell
(venv) [t0xic0der@fedorable protop2g]$ protop2g -g xxxxxxxxxxxxxxxxxx -p xxxxxxxxxxxxxxxxxx -s mdapi -d 42823949 -f t0xic0der -t t0xic0der repo
[ BUSY ] Requesting for source namespace metadata...
[ PASS ] Source namespace metadata acquisition succeeded!
Name: mdapi
Identifier: 221
Maintainer: Pierre-YvesChibon (ID pingou)
Location: https://pagure.io/mdapi
Address: https://t0xic0der:xxxxxxxxxxxxxxxxxx@pagure.io/mdapi.git
Created on: Thu Oct 22 16:33:58 2015
Last modified on: Thu Oct 22 16:33:58 2015
Tags: ['fedmsg', 'fedora-infra']
[ BUSY ] Requesting for destination namespace metadata...
[ PASS ] Destination namespace metadata acquisition succeeded!
Name: t0xic0der/protop2g-test
Identifier: 42823949
Maintainer: Akashdeep Dhar (ID t0xic0der)
Location: https://gitlab.com/t0xic0der/protop2g-test
Address: https://t0xic0der:xxxxxxxxxxxxxxxxxx@gitlab.com/t0xic0der/protop2g-test.git
Created on: 2023-01-23T16:18:30.217Z
Last modified on: 2023-01-28T18:44:14.075Z
Tags: []
[ BUSY ] Attempting source namespace assets clone...
[ PASS ] Source namespace assets clone succeeded!
Directory: /var/tmp/protop2g-tempsrce-a63xrvkc
Time taken: 6.77 second(s)
[ BUSY ] Attempting destination namespace assets clone...
[ PASS ] Destination namespace assets clone succeeded!
Directory: /var/tmp/protop2g-tempdest-w6_kxcbz
Time taken: 2.36 second(s)
[ BUSY ] Reading branches data from the locally cloned assets...
[ PASS ] Branches data reading succeeded!
Available in source namespace: 4 branch(es)
- (SRCE branch) HEAD
- (SRCE branch) master
- (SRCE branch) production
- (SRCE branch) staging
Available in destination namespace: 5 branch(es)
- (DEST branch) HEAD
- (DEST branch) main
- (DEST branch) master
- (DEST branch) production
- (DEST branch) staging
Requested for transferring: 0 branch(es)
[ BUSY ] Initializing namespace assets transfer...
[ WARN ] Transferring 4 available branches
[1/4] Branch 'HEAD' was transferred to the destination namespace
[2/4] Branch 'master' was transferred to the destination namespace
[3/4] Branch 'production' was transferred to the destination namespace
[4/4] Branch 'staging' was transferred to the destination namespace
Assets transferred: 4 branch(es) completed, 4 branch(es) requested
Time taken: 4.99 second(s)
[ PASS ] Namespace assets transfer succeeded!
.. code-block:: shell
(venv) [t0xic0der@fedorable protop2g]$ protop2g -g xxxxxxxxxxxxxxxxxx -p xxxxxxxxxxxxxxxxxx -s mdapi -d 42823949 -f t0xic0der -t t0xic0der repo
[ BUSY ] Requesting for source namespace metadata...
[ PASS ] Source namespace metadata acquisition succeeded!
Name: mdapi
Identifier: 221
Maintainer: Pierre-YvesChibon (ID pingou)
Location: https://pagure.io/mdapi
Address: https://t0xic0der:xxxxxxxxxxxxxxxxxx@pagure.io/mdapi.git
Created on: Thu Oct 22 16:33:58 2015
Last modified on: Thu Oct 22 16:33:58 2015
Tags: ['fedmsg', 'fedora-infra']
[ BUSY ] Requesting for destination namespace metadata...
[ PASS ] Destination namespace metadata acquisition succeeded!
Name: t0xic0der/protop2g-test
Identifier: 42823949
Maintainer: Akashdeep Dhar (ID t0xic0der)
Location: https://gitlab.com/t0xic0der/protop2g-test
Address: https://t0xic0der:xxxxxxxxxxxxxxxxxx@gitlab.com/t0xic0der/protop2g-test.git
Created on: 2023-01-23T16:18:30.217Z
Last modified on: 2023-01-28T18:44:14.075Z
Tags: []
[ BUSY ] Attempting source namespace assets clone...
[ PASS ] Source namespace assets clone succeeded!
Directory: /var/tmp/protop2g-tempsrce-a63xrvkc
Time taken: 6.77 second(s)
[ BUSY ] Attempting destination namespace assets clone...
[ PASS ] Destination namespace assets clone succeeded!
Directory: /var/tmp/protop2g-tempdest-w6_kxcbz
Time taken: 2.36 second(s)
[ BUSY ] Reading branches data from the locally cloned assets...
[ PASS ] Branches data reading succeeded!
Available in source namespace: 4 branch(es)
- (SRCE branch) HEAD
- (SRCE branch) master
- (SRCE branch) production
- (SRCE branch) staging
Available in destination namespace: 5 branch(es)
- (DEST branch) HEAD
- (DEST branch) main
- (DEST branch) master
- (DEST branch) production
- (DEST branch) staging
Requested for transferring: 0 branch(es)
[ BUSY ] Initializing namespace assets transfer...
[ WARN ] Transferring 4 available branches
[1/4] Branch 'HEAD' was transferred to the destination namespace
[2/4] Branch 'master' was transferred to the destination namespace
[3/4] Branch 'production' was transferred to the destination namespace
[4/4] Branch 'staging' was transferred to the destination namespace
Assets transferred: 4 branch(es) completed, 4 branch(es) requested
Time taken: 4.99 second(s)
[ PASS ] Namespace assets transfer succeeded!
Resources
----
---------
The project can be found over `here <https://github.com/t0xic0der/protop2g>`_.
The project can be found over `here <https://github.com/t0xic0der/protop2g>`__.

321
docs/pdc/endpoints.rst
View file

@ -1,195 +1,194 @@
Endpoints
=========
Based on the users we identified, we have identified the endpoints our scripts and services are actually using.
Based on the users we identified, we have identified the endpoints our scripts and
services are actually using.
/rest_api/v1/global-components/?name={{name}}
-----------------------
---------------------------------------------
::
.. code-block::
{
"dist_git_path": null, # dont use these
"dist_git_web_url": "https://src.fedoraproject.org/cgit/rpms/gnupg2.git", # generated from name
"id": 20,
"labels": [], # we don't use these
"name": "gnupg2",
"upstream": null # we don't use this
}
{
"dist_git_path": null, # dont use these
"dist_git_web_url": "https://src.fedoraproject.org/cgit/rpms/gnupg2.git", # generated from name
"id": 20,
"labels": [], # we don't use these
"name": "gnupg2",
"upstream": null # we don't use this
}
/rest_api/v1/component-branches/
--------------------
--------------------------------
::
.. code-block::
{
"active": true, # I don't know where do we store this?
"critical_path": false,
"global_component": "0ad",
"id": 501964,
"name": "f34",
"slas": [
{
"eol": "2022-03-08",
"id": 1003101,
"sla": "security_fixes"
},
{
"eol": "2022-03-08",
"id": 1003102,
"sla": "bug_fixes"
}
],
"type": "rpm" # | rpm | container | zip | iso | composite | module | flatpak
{
"active": true, # I don't know where do we store this?
"critical_path": false,
"global_component": "0ad",
"id": 501964,
"name": "f34",
"slas": [
{
"eol": "2022-03-08",
"id": 1003101,
"sla": "security_fixes"
},
{
"eol": "2022-03-08",
"id": 1003102,
"sla": "bug_fixes"
}
],
"type": "rpm" # | rpm | container | zip | iso | composite | module | flatpak
}
rest_api/v1/component-branch-slas/
----------------------
----------------------------------
Couldn't find the shape without authenticating
rest_api/v1/compose-images/
---------------
---------------------------
stores `composeinfo.json` and `image-manifest.json`
gives ability to retrieve list of images in specified compose
stores `composeinfo.json` and `image-manifest.json` gives ability to retrieve list
of images in specified compose
::
.. code-block::
{
"header": {
"type": "productmd.images",
"version": "1.2"
},
"payload": {
"compose": {
"date": "20160416",
"id": "Fedora-Rawhide-20160416.n.0",
"respin": 0,
"type": "nightly"
{
"header": {
"type": "productmd.images",
"version": "1.2"
},
"images": {
"Workstation": {
"x86_64": [
{
"arch": "x86_64",
"bootable": true,
"checksums": {
"sha256": "a7cbd69cc5c96dbf0e0f39d29b09c27844af677a602f512672527a5f7a91000c"
},
"disc_count": 1,
"disc_number": 1,
"format": "iso",
"implant_md5": "ea67ef79f3374d91735008148de4146e",
"mtime": 1460789082,
"path": "Workstation/x86_64/iso/Fedora-Workstation-netinst-x86_64-Rawhide-20160416.n.0.iso",
"size": 426770432,
"subvariant": "Workstation",
"type": "boot",
"volume_id": "Fedora-WS-dvd-x86_64-rawh"
}
]
"payload": {
"compose": {
"date": "20160416",
"id": "Fedora-Rawhide-20160416.n.0",
"respin": 0,
"type": "nightly"
},
"images": {
"Workstation": {
"x86_64": [
{
"arch": "x86_64",
"bootable": true,
"checksums": {
"sha256": "a7cbd69cc5c96dbf0e0f39d29b09c27844af677a602f512672527a5f7a91000c"
},
"disc_count": 1,
"disc_number": 1,
"format": "iso",
"implant_md5": "ea67ef79f3374d91735008148de4146e",
"mtime": 1460789082,
"path": "Workstation/x86_64/iso/Fedora-Workstation-netinst-x86_64-Rawhide-20160416.n.0.iso",
"size": 426770432,
"subvariant": "Workstation",
"type": "boot",
"volume_id": "Fedora-WS-dvd-x86_64-rawh"
}
]
}
}
}
}
}
/rest_api/v1/releases
-----------------------
/rest_api/v1/releases
---------------------
::
.. code-block::
{
"active": false,
"allow_buildroot_push": false,
"allowed_debuginfo_services": [],
"allowed_push_targets": [],
"base_product": null,
"bugzilla": null,
"compose_set": [],
"dist_git": {
"branch": "f25"
{
"active": false,
"allow_buildroot_push": false,
"allowed_debuginfo_services": [],
"allowed_push_targets": [],
"base_product": null,
"bugzilla": null,
"compose_set": [],
"dist_git": {
"branch": "f25"
},
"integrated_with": null,
"name": "Fedora Updates",
"product_version": "fedora-25",
"release_id": "fedora-25-updates",
"release_type": "updates",
"short": "fedora",
"sigkey": null,
"version": "25"
},
"integrated_with": null,
"name": "Fedora Updates",
"product_version": "fedora-25",
"release_id": "fedora-25-updates",
"release_type": "updates",
"short": "fedora",
"sigkey": null,
"version": "25"
},
rest_api/v1/rpms
---------------
::
{
"count": 12473109,
"next": "https://pdc.fedoraproject.org/rest_api/v1/rpms/?page=2",
"previous": null,
"results": [
{
"arch": "x86_64",
"built_for_release": null,
"dependencies": {
"conflicts": [],
"obsoletes": [],
"provides": [],
"recommends": [],
"requires": [],
"suggests": []
},
"epoch": 0,
"filename": "readline-6.3-7.fc24.x86_64.rpm",
"id": 20,
"linked_composes": [
"Fedora-24-20160202.n.0",
"Fedora-24-20160203.n.0",
"Fedora-24-20160203.n.1",
"Fedora-24-20160204.n.0"
],
"linked_releases": [],
"name": "readline",
"release": "7.fc24",
"srpm_commit_branch": null,
"srpm_commit_hash": null,
"srpm_name": "readline",
"srpm_nevra": "readline-0:6.3-7.fc24.src",
"version": "6.3"
} ] }
/rest_api/v1/product-versions
----------------
::
{
"active": true,
"allowed_push_targets": [],
"name": "fedora 34",
"product": "fedora",
"product_version_id": "fedora-34",
"releases": [
"fedora-34"
],
"short": "fedora",
"version": "34"
},
{
"active": true,
"allowed_push_targets": [],
"name": "fedora rawhide",
"product": "fedora",
"product_version_id": "fedora-rawhide",
"releases": [
"fedora-Rawhide",
"fedora-modular-Rawhide"
],
"short": "fedora",
"version": "rawhide"
}
.. code-block::
{
"count": 12473109,
"next": "https://pdc.fedoraproject.org/rest_api/v1/rpms/?page=2",
"previous": null,
"results": [
{
"arch": "x86_64",
"built_for_release": null,
"dependencies": {
"conflicts": [],
"obsoletes": [],
"provides": [],
"recommends": [],
"requires": [],
"suggests": []
},
"epoch": 0,
"filename": "readline-6.3-7.fc24.x86_64.rpm",
"id": 20,
"linked_composes": [
"Fedora-24-20160202.n.0",
"Fedora-24-20160203.n.0",
"Fedora-24-20160203.n.1",
"Fedora-24-20160204.n.0"
],
"linked_releases": [],
"name": "readline",
"release": "7.fc24",
"srpm_commit_branch": null,
"srpm_commit_hash": null,
"srpm_name": "readline",
"srpm_nevra": "readline-0:6.3-7.fc24.src",
"version": "6.3"
} ] }
/rest_api/v1/product-versions
-----------------------------
.. code-block::
{
"active": true,
"allowed_push_targets": [],
"name": "fedora 34",
"product": "fedora",
"product_version_id": "fedora-34",
"releases": [
"fedora-34"
],
"short": "fedora",
"version": "34"
},
{
"active": true,
"allowed_push_targets": [],
"name": "fedora rawhide",
"product": "fedora",
"product_version_id": "fedora-rawhide",
"releases": [
"fedora-Rawhide",
"fedora-modular-Rawhide"
],
"short": "fedora",
"version": "rawhide"
}

83
docs/pdc/index.rst
View file

@ -1,23 +1,26 @@
PDC replacement research
========================
`PDC <https://pdc.fedoraproject.org/>`_ is repository and API for storing and querying metadata related to packages and product releases.
We rely on information in it to produce Fedora releases, manage retirement of packages and more.
`PDC <https://pdc.fedoraproject.org/>`_ is repository and API for storing and querying
metadata related to packages and product releases. We rely on information in it to
produce Fedora releases, manage retirement of packages and more.
Our current deployment running on https://pdc.fedoraproject.org/.
It uses https://github.com/product-definition-center/product-definition-center/tree/python-pdc-1.9.0-2
It uses
https://github.com/product-definition-center/product-definition-center/tree/python-pdc-1.9.0-2
The software is orphaned by it's original maintainers with version requiring EOL version of python and EOL version of Django Framework,
this means we need to upgrade it or replace it.
The software is orphaned by it's original maintainers with version requiring EOL version
of python and EOL version of Django Framework, this means we need to upgrade it or
replace it.
Abstract take-away
------------------
There is no silver bullet, it will require development effort
Responsibility for retirement and sla's of packages could be moved to pagure-dist_git
Tesponsibility for critical-path and releases can be moved to Bodhi
Nightly compose metadata requires new application, that would be much simpler than current PDC
There is no silver bullet, it will require development effort Responsibility for
retirement and sla's of packages could be moved to pagure-dist_git Tesponsibility for
critical-path and releases can be moved to Bodhi Nightly compose metadata requires new
application, that would be much simpler than current PDC
.. toctree::
:maxdepth: 1
@ -29,28 +32,31 @@ Nightly compose metadata requires new application, that would be much simpler th
Previous work
-------------
There has been `a proposal by Clement Verna <https://docs.google.com/document/d/1agV5YRtDvqE2UxLl9f6E_l0iukRkv7F4qCXAx4PfFng/edit#>`_
with much of the usecases already mapped out, and `a POC in fedora-infra/fpdc repo <https://github.com/fedora-infra/fpdc/tree/master/fpdc>`_ ,
but it was left for more pressing work.
There has been `a proposal by Clement Verna
<https://docs.google.com/document/d/1agV5YRtDvqE2UxLl9f6E_l0iukRkv7F4qCXAx4PfFng/edit#>`_
with much of the usecases already mapped out, and `a POC in fedora-infra/fpdc repo
<https://github.com/fedora-infra/fpdc/tree/master/fpdc>`_ , but it was left for more
pressing work.
Current users of PDC
--------------------
Based on conversations with coleagues from Releng and Fedora QA,
and cursory investigation of our repositories we use pdc API or through CLI client in:
Based on conversations with coleagues from Releng and Fedora QA, and cursory
investigation of our repositories we use pdc API or through CLI client in:
- releng scripts
- fedpkg
- fedfind
- bodhi
- pagure
- modulebuildservice
- modulebuildservice
- mirrormanager scripts in ansible-repo
- new hotness
- fedora messaging
- fedora messaging
- osbs client
As a part of this investigation we want to create exhaustive list, with analysis of the actual PDC use-case in each application.
As a part of this investigation we want to create exhaustive list, with analysis of the
actual PDC use-case in each application.
Solutions to be explored
------------------------
@ -60,15 +66,16 @@ We have sevral options:
- Upgrade PDC to supported version of Django and take over the maintenance
- Use a database with a simple api gateway like Postgrest or Prest in front of it
- Create a new bespoke application that better suits our needs
- Incorporate the functionality we need to other applications we already have (Pagure/Bodhi/Datagrepper)
- Incorporate the functionality we need to other applications we already have
(Pagure/Bodhi/Datagrepper)
Currently we are proposing primarily the last option.
Preliminary notes on maintianing current PDC
--------------------------------
--------------------------------------------
This would require significant investment, as it would require several upgrades of underlying software
and probably even a Python 2.x to 3.x migration as well.
This would require significant investment, as it would require several upgrades of
underlying software and probably even a Python 2.x to 3.x migration as well.
Moreover, based on the discussions within CPE:
@ -78,32 +85,30 @@ Moreover, based on the discussions within CPE:
Because of this, we won't focus on this avenue of exploration.
Preliminary notes on using Postgrest
-------------------------------------
------------------------------------
Based on our discussions, we mostly use simple CRUD API,
which means that a new database-model that would better suit our needs
could be all we need to migrate.
Based on our discussions, we mostly use simple CRUD API, which means that a new
database-model that would better suit our needs could be all we need to migrate.
The biggest advantage of this approach is using off the shelf service,
where only thing we need to maintain is the database-model, clients,
and keeping the software up-to-date.
The biggest advantage of this approach is using off the shelf service, where only thing
we need to maintain is the database-model, clients, and keeping the software up-to-date.
We have chosen two candidates to investigate based on the fact,
we probably want to utilize Postgresql on the backend:
We have chosen two candidates to investigate based on the fact, we probably want to
utilize Postgresql on the backend:
- https://postgrest.org/en/stable/
Postgrest as a simple api on top of a database model sounded promissing,
but after investigating how to integrate it with ou currentaccount system,
it turned out we would need to maintain a separate service that would serve as a bridge.
- https://postgrest.org/en/stable/auth.html
- https://samkhawase.com/blog/postgrest/postgrest_auth0_service/
Postgrest as a simple api on top of a database model sounded promissing, but after
investigating how to integrate it with ou currentaccount system, it turned out we would
need to maintain a separate service that would serve as a bridge. -
https://postgrest.org/en/stable/auth.html -
https://samkhawase.com/blog/postgrest/postgrest_auth0_service/
This speaks against using Postgrest.
Create a new bespoke application that better suits our needs
----------------------------------------------------
------------------------------------------------------------
We investigated using fastapi and fas, and the previously started fpdc proof of concept by Clement.
Based on discussions with others, we decided to investigate way to merge the existing functionality to the services we already maintain.
We investigated using fastapi and fas, and the previously started fpdc proof of concept
by Clement. Based on discussions with others, we decided to investigate way to merge the
existing functionality to the services we already maintain.

57
docs/pdc/split.rst
View file

@ -1,5 +1,5 @@
Proposal to retire PDC by splitting it among existing services
====================
==============================================================
Based on our research, currently we are using PDC for:
@ -10,10 +10,12 @@ Based on our research, currently we are using PDC for:
The endpoints we use are:
- /rest_api/v1/global-components - for storing package names
- /rest_api/v1/component-branches/ - stores SLA's, active/retired and critical-path flag
- /rest_api/v1/component-branches/ - stores SLA's, active/retired and critical-path flag
- /rest_api/v1/component-branch-slas/ - auxiliary endpoint for SLA's
- /rest_api/v1/compose-images//rest_api/v1/compose-images/ - stores composeinfo.json and image-manifest.json
- /rest_api/v1/releases/ - metadata on type (ga, updates, eus, aus, els, fast, updates-testing) of a release of a product version and if it is active
- /rest_api/v1/compose-images//rest_api/v1/compose-images/ - stores composeinfo.json and
image-manifest.json
- /rest_api/v1/releases/ - metadata on type (ga, updates, eus, aus, els, fast,
updates-testing) of a release of a product version and if it is active
- /rest_api/v1/rpms/ - liks rpms to composes
- /rest_api/v1/product-versions - stores major versions of fedora
@ -21,47 +23,50 @@ Bodhi
-----
Bodhi would take over responsibilities of
- /rest_api/v1/releases/
- /rest_api/v1/component-branches/ - partially, just the critical-path flag
- /rest_api/v1/releases/
- /rest_api/v1/component-branches/ - partially, just the critical-path flag
Bodhi already has concept that maps to releases and components,
we should be able to create auxiliary table that will pair these with additional metadata,
mostyl the critical-path flag, that previously had to be queried from PDC.
Bodhi already has concept that maps to releases and components, we should be able to
create auxiliary table that will pair these with additional metadata, mostyl the
critical-path flag, that previously had to be queried from PDC.
PDC actually isn't the source-of-truth for packages on critical path.
The source of truth, according to https://fedoraproject.org/wiki/Critical_path_package is
in https://pagure.io/fedora-comps/tree/main when you search for groups with critical-path-*
PDC actually isn't the source-of-truth for packages on critical path. The source of
truth, according to https://fedoraproject.org/wiki/Critical_path_package is in
https://pagure.io/fedora-comps/tree/main when you search for groups with critical-path-*
prefix. For example: https://pagure.io/fedora-comps/blob/main/f/comps-f35.xml.in#_622
The data is available through dnf, but generating it can be really slow, PDC is then used as
a sort of a pre-computed cache.
The data is available through dnf, but generating it can be really slow, PDC is then
used as a sort of a pre-computed cache.
To update it, we used to use https://docs.pagure.org/releng/sop_update_critpath.html
but now we have https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/plugins/pdc_update_critpath.py
To update it, we used to use https://docs.pagure.org/releng/sop_update_critpath.html but
now we have
https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/plugins/pdc_update_critpath.py
Only place we actually use the information during the fedora release is in Bodhi,
to have stricter requirements on `critpath` package updates in Bodhi:
Only place we actually use the information during the fedora release is in Bodhi, to
have stricter requirements on `critpath` package updates in Bodhi:
https://github.com/fedora-infra/bodhi/blob/master/bodhi/server/util.py#L192
Pagure-dist-git
---------------
Pagure-dist-git would take over responsibilities for
Pagure-dist-git would take over responsibilities for
- /rest_api/v1/product-versions
- /rest_api/v1/global-components
- /rest_api/v1/component-branches/
- /rest_api/v1/component-branch-slas/
Pagure already de-facto has a database of global-components (the repositories), and product-versions (repository branches)
and it uses PDC api to query component-branches if the package was retired,
with auxiliary table in pagure-dist-git storing reason for orphanning.
Pagure already de-facto has a database of global-components (the repositories), and
product-versions (repository branches) and it uses PDC api to query component-branches
if the package was retired, with auxiliary table in pagure-dist-git storing reason for
orphanning.
Remaining endpoints
-------------
-------------------
/rest_api/v1/compose-images/ - as we are mostly storing json blobs, based on discussions, we should just store the json in network-accessible file-server.
/rest_api/v1/compose-images/ - as we are mostly storing json blobs, based on
discussions, we should just store the json in network-accessible file-server.
/rest_api/v1/rpms/ - this is only information that so far doesn't fit into any existing application. QA is using it to query what packages are in particular composes.
/rest_api/v1/rpms/ - this is only information that so far doesn't fit into any existing
application. QA is using it to query what packages are in particular composes.

92
docs/pdc/users.rst
View file

@ -2,32 +2,32 @@ Current users of PDC
====================
release engineering - releng scripts
-----------------------------
------------------------------------
Repository: https://pagure.io/releng
Retired packages
- scripts/block_retired.py
- /rest_api/v1/component-branches/?name={1}&type={2}&active=false&page_size=100'
- {1} - branch name, i.e. rawhide
- {2} - type, i.e. rpm
- scripts/block_retired.py
- /rest_api/v1/component-branches/?name={1}&type={2}&active=false&page_size=100'
- {1} - branch name, i.e. rawhide
- {2} - type, i.e. rpm
Package unretirement
- scripts/check-unretirement.py
- https://pdc.fedoraproject.org/rest_api/v1/component-branches/?global_component={args.pck}&name={args.brc}&type={args.nms[:-1]}
- scripts/check-unretirement.py
- https://pdc.fedoraproject.org/rest_api/v1/component-branches/?global_component={args.pck}&name={args.brc}&type={args.nms[:-1]}
Package retirement
- scripts/fedretire
- results = pdc.get_paged(pdc['component-branches'], global_component=name, name=branch, type=NAMESPACES[namespace])
- uses fedpkg for actual retirement
- scripts/fedretire
- results = pdc.get_paged(pdc['component-branches'], global_component=name,
name=branch, type=NAMESPACES[namespace])
- uses fedpkg for actual retirement
Critical path
- scripts/get-critpath
- pdc.get_paged(endpoint, critical_path=True, **kwargs)
- scripts/get-critpath
- ``pdc.get_paged(endpoint, critical_path=True, **kwargs)``
Mass rebuild modules
- /rest_api/v1/component-branch-slas/?page_size=100&branch_type=module&branch_active=1'
- /rest_api/v1/component-branch-slas/?page_size=100&branch_type=module&branch_active=1'
fedpkg
------
@ -35,26 +35,26 @@ fedpkg
Repository: https://pagure.io/fedpkg
get_release_branches
- fedpkg/utils.py
- query_args = {'fields': ['short', 'version'], 'active': True}
- endpoint = 'product-versions'
- fedpkg/utils.py
- query_args = {'fields': ['short', 'version'], 'active': True}
- endpoint = 'product-versions'
get_stream_branches
- fedpkg/utils.py
- query_args = {'global_component': package_name,'fields': ['name', 'active']}
- endpoint = 'component-branches'
- fedpkg/utils.py
- query_args = {'global_component': package_name,'fields': ['name', 'active']}
- endpoint = 'component-branches'
query_pdc
- fedpkg/utils.py
- uses requests
- fedpkg/utils.py
- uses requests
pungi
----------
-----
According to lubomir.sedlar@gmail.com:
"Pungi used to integrate with PDC in the past, but this support has been
deprecated and dropped."
"Pungi used to integrate with PDC in the past, but this support has been deprecated and
dropped."
fedoraqa - fedfind
------------------
@ -78,7 +78,7 @@ Pungi4Release.get_package_nevras_pdc
- pdc_query('rpms', [('compose', self.cid), ('arch', 'src')])
Bodhi
------------------
-----
Repository: https://github.com/fedora-infra/bodhi
@ -91,14 +91,15 @@ get_critpath_components_from_pdc
fedscm-admin
------------
We have dropped fedscm-admin in favor of toddlers.
The endpoints used by it should be the same since it solves the same problem.
We have dropped fedscm-admin in favor of toddlers. The endpoints used by it should be
the same since it solves the same problem.
Repository: https://pagure.io/fedora-infra/toddlers/
fedscm_admin/pdc.py
fedscm_admin/pdc.py
- get_global_component
- new_global_component
- new_global_component
- get_branch
- new_branch
- get_sla
@ -106,23 +107,26 @@ fedscm_admin/pdc.py
pagure
------
- /rest_api/v1/component-branches/
- to get retired packages
- /rest_api/v1/component-branches/
- to get retired packages
modulebuildservice
------------------
As of Fedora 39 we are dropping modules support. With Fedora 38 EOL we will drop MBS.
- /rest_api/v1/component-branches/
- to get end-of-life packages
- based on ansible roles/mbs/common/files/fedora.json.productbion
- /rest_api/v1/component-branches/
- to get end-of-life packages
- based on ansible roles/mbs/common/files/fedora.json.productbion
mirrormanager scripts
---------------------
- /rest_api/v1/releases/
- checks for currently active releases
- roles/mirrormanager/backend/templates/handle_propagation.sh
- roles/mirrormanager/crawler/files/check_propagation.sh
- /rest_api/v1/releases/
- checks for currently active releases
- roles/mirrormanager/backend/templates/handle_propagation.sh
- roles/mirrormanager/crawler/files/check_propagation.sh
ODCS
----
@ -131,16 +135,16 @@ no longer uses pdc
new hotness
-----------
- /rest_api/v1/component-branches/
- to get retired packages
- /rest_api/v1/component-branches/
- to get retired packages
fedora messaging
----------------
- no integrations found
osbs client
-----------
- no integrations found

72
docs/registry_to_quay/fedora_images.rst
View file

@ -3,22 +3,23 @@
Fedora Base/Minimal/CoreOS container images
===========================================
This document contains information about current workflow used to build and publish these container
images. It also describes changes needed to make it work with `quay.io <https://quay.io>`_.
This document contains information about current workflow used to build and publish
these container images. It also describes changes needed to make it work with `quay.io
<https://quay.io>`_.
Current workflow
----------------
The base/minimal/CoreOS images are currently built using OSBS. The OSBS submits builds to
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_ using
dockercfg and user can create an update in Bodhi. After testing period is finished or the update
gets enough karma it is moved by bodhi to
`registry.fedoraproject_org <https://registry.fedoraproject.org/>`_.
The base/minimal/CoreOS images are currently built using OSBS. The OSBS submits builds
to `candidate-registry.fedoraproject.org
<https://candidate-registry.fedoraproject.org/>`_ using dockercfg and user can create an
update in Bodhi. After testing period is finished or the update gets enough karma it is
moved by bodhi to `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_.
This will change in F39 where the images will be built by
`koji-flatpak <https://pagure.io/koji-flatpak>`_ plugin. Releng script will then submit the builds
to both `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and
`quay.io <https://quay.io>`_.
This will change in F39 where the images will be built by `koji-flatpak
<https://pagure.io/koji-flatpak>`_ plugin. Releng script will then submit the builds to
both `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and `quay.io
<https://quay.io>`_.
.. image:: ../_static/container_workflow_registry.drawio.png
:target: ../_images/container_workflow_registry.drawio.png
@ -31,37 +32,40 @@ work with current setup. It will go through pipeline step by step and describe w
needs to be changed in case of base/minimal/CoreOS images workflow.
OSBS
^^^^
~~~~
OSBS is using Openshift local container registry, which is configured by
`dockercfg <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for
`production <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_ and `staging <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in
`setup-worker-namespace <https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_ playbook.
To make this work with `quay.io <https://quay.io>`_ we need to replace the dockercfg with new
one pointing to `quay.io <https://quay.io>`_.
OSBS is using Openshift local container registry, which is configured by `dockercfg
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for `production
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
and `staging
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in `setup-worker-namespace
<https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_
playbook. To make this work with `quay.io <https://quay.io>`_ we need to replace the
dockercfg with new one pointing to `quay.io <https://quay.io>`_.
.. note::
OSBS could have trouble reaching to `quay.io <https://quay.io>`_, this needs to be solved
for this to work.
OSBS could have trouble reaching to `quay.io <https://quay.io>`_, this needs to be
solved for this to work.
Releng script
^^^^^^^^^^^^^
~~~~~~~~~~~~~
In the new workflow introduced in Fedora 39+ is a Release Engineering script that pushes the image
to registry. This script could be found in
`releng repository <https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh>`_
and it's already pushing images to `quay.io <https://quay.io>`_.
In the new workflow introduced in Fedora 39+ is a Release Engineering script that pushes
the image to registry. This script could be found in `releng repository
<https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh>`_ and
it's already pushing images to `quay.io <https://quay.io>`_.
Bodhi
^^^^^
~~~~~
Bodhi is pushing the container builds from
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_ to
`registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and this is configured in
`production.ini template <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
Bodhi is pushing the container builds from `candidate-registry.fedoraproject.org
<https://candidate-registry.fedoraproject.org/>`_ to `registry.fedoraproject_org
<https://registry.fedoraproject.org/>`_ and this is configured in `production.ini
template
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
in ansible.
To make it work with quay.io we just need to change `container.destination_registry` and
@ -69,5 +73,5 @@ To make it work with quay.io we just need to change `container.destination_regis
.. note::
Bodhi wouldn't be part of the base/minimal/CoreOS images workflow in Fedora 39 and later.\
So no need to change anything here.
Bodhi wouldn't be part of the base/minimal/CoreOS images workflow in Fedora 39 and
later.So no need to change anything here.

90
docs/registry_to_quay/flatpaks.rst
View file

@ -3,20 +3,21 @@
Fedora Flatpaks
===============
This document contains information about current workflow used to build and publish Fedora flatpaks.
It also describes changes needed to make it work with `quay.io <https://quay.io>`_.
This document contains information about current workflow used to build and publish
Fedora flatpaks. It also describes changes needed to make it work with `quay.io
<https://quay.io>`_.
Flatpaks are the biggest tenant occupying `registry.fedoraproject.org <https://registry.fedoraproject.org/>`_
right now.
Flatpaks are the biggest tenant occupying `registry.fedoraproject.org
<https://registry.fedoraproject.org/>`_ right now.
Current workflow
----------------
The flatpaks are build in koji using OSBS. The OSBS then submits the build to
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_
and packager can create an update in Bodhi. After testing period is finished or the update
gets enough karma it is moved by bodhi to
`registry.fedoraproject_org <https://registry.fedoraproject.org/>`_.
and packager can create an update in Bodhi. After testing period is finished or the
update gets enough karma it is moved by bodhi to `registry.fedoraproject_org
<https://registry.fedoraproject.org/>`_.
.. image:: ../_static/flatpaks_workflow_registry.drawio.png
:target: ../_images/flatpaks_workflow_registry.drawio.png
@ -24,61 +25,68 @@ gets enough karma it is moved by bodhi to
Changes for quay.io
-------------------
This section will describe changes that are needed to make quay.io work with current setup.
It will go through pipeline step by step and describe what needs to be changed in case of
flatpaks workflow.
This section will describe changes that are needed to make quay.io work with current
setup. It will go through pipeline step by step and describe what needs to be changed in
case of flatpaks workflow.
OSBS
^^^^
OSBS is using Openshift local container registry, which is configured by
`dockercfg <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for
`production <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_ and `staging <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in
`setup-worker-namespace <https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_ playbook.
To make this work with quay.io we need to replace the dockercfg with new one pointing to quay.io.
~~~~
OSBS is using Openshift local container registry, which is configured by `dockercfg
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for `production
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
and `staging
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in `setup-worker-namespace
<https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_
playbook. To make this work with quay.io we need to replace the dockercfg with new one
pointing to quay.io.
.. note::
OSBS could have trouble reaching to quay.io, this needs to be solved for this to work.
OSBS could have trouble reaching to quay.io, this needs to be solved for this to
work.
Koji
^^^^
There is currently a initiative that is trying to remove OSBS from flatpak build pipeline. At
the time of writing this document this is only enabled for F39. But as the F38 flaptak runtime
will be considered EOL the `koji-flatpak <https://pagure.io/koji-flatpak>`_ module will be the only
way.
~~~~
The koji-flatpak module is much direct, there is just one
`configuration file <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/koji_builder/templates/flatpak.conf>`_
used to specify where the koji builders should push the build container. The `candidate-registry`
variable is set in `tasks/main`.
There is currently a initiative that is trying to remove OSBS from flatpak build
pipeline. At the time of writing this document this is only enabled for F39. But as the
F38 flaptak runtime will be considered EOL the `koji-flatpak
<https://pagure.io/koji-flatpak>`_ module will be the only way.
Currently the builders don't have access to quay.io. The firewall settings needs to be adjusted
so they could push containers.
The koji-flatpak module is much direct, there is just one `configuration file
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/koji_builder/templates/flatpak.conf>`_
used to specify where the koji builders should push the build container. The
`candidate-registry` variable is set in `tasks/main`.
Currently the builders don't have access to quay.io. The firewall settings needs to be
adjusted so they could push containers.
Bodhi
^^^^^
~~~~~
Bodhi is pushing the container builds from candidate-registry.fedoraproject.org to
registry.fedoraproject.org and this is configured in
`production.ini template <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
registry.fedoraproject.org and this is configured in `production.ini template
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
in ansible.
To make it work with quay.io we just need to change `container.destination_registry` and
`container.source_registry` variables.
Proxies
^^^^^^^
~~~~~~~
We need to redirect all the pull requests for registry.fedoraproject.org to quay.io. This could be
done by `mod_proxy / mod_rewrite` and redirect all requests to
We need to redirect all the pull requests for registry.fedoraproject.org to quay.io.
This could be done by `mod_proxy / mod_rewrite` and redirect all requests to
https://registry.fedoraproject.org/v2/REPO/ to https://quay/v2/fedora/REPO/
flatpak-indexer
^^^^^^^^^^^^^^^
~~~~~~~~~~~~~~~
`flatpak-indexer <https://github.com/owtaylor/flatpak-indexer>`_ is used to read metadata from containers
and generate index in format flatpak client understands. It also creates deltas for updates. It's deployed in OpenShift
and the changes have to be done in
`configuration <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/flatpak-indexer/templates/configmap.yml>`_.
`flatpak-indexer <https://github.com/owtaylor/flatpak-indexer>`_ is used to read
metadata from containers and generate index in format flatpak client understands. It
also creates deltas for updates. It's deployed in OpenShift and the changes have to be
done in `configuration
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/flatpak-indexer/templates/configmap.yml>`_.

68
docs/registry_to_quay/index.rst
View file

@ -4,48 +4,56 @@ Migrate registry.fedoraproject.org to quay.io
Purpose
-------
This investigation is looking into service `quay.io <https://quay.io/>`_ and how can we utilize
it for the projects hosted on `registry.fedoraproject.org <https://registry.fedoraproject.org/>`_. It should also resolve if this is work should be initiative.
This investigation is looking into service `quay.io <https://quay.io/>`_ and how can we
utilize it for the projects hosted on `registry.fedoraproject.org
<https://registry.fedoraproject.org/>`_. It should also resolve if this is work should
be initiative.
Resources
---------
* Fedora Infrastructure ticket: https://pagure.io/fedora-infrastructure/issue/10386
- Fedora Infrastructure ticket: https://pagure.io/fedora-infrastructure/issue/10386
Requirements
------------
* Multi Arch containers (Already supported on quay.io)
* OCI images (Already supported on quay.io)
* Web interface (Quay.io already has web interface)
* Integrate Quay.io in current workflow
* Must not affect users of images
* `candidate-registry <https://candidate-registry.fedoraproject.org/>`_ must be moved as well
- Multi Arch containers (Already supported on quay.io)
- OCI images (Already supported on quay.io)
- Web interface (Quay.io already has web interface)
- Integrate Quay.io in current workflow
- Must not affect users of images
- `candidate-registry <https://candidate-registry.fedoraproject.org/>`_ must be moved as
well
Nice to have
------------
* Staging namespace on quay.io
- Staging namespace on quay.io
Risks
-----
* There is a `API endpoint throttling <https://docs.quay.io/issues/429.html>`_ per second on IP address on quay.io. This could cause issues in the future, but current load should be easily handled.
- There is a `API endpoint throttling <https://docs.quay.io/issues/429.html>`_ per
second on IP address on quay.io. This could cause issues in the future, but current
load should be easily handled.
Statistics
----------
These statistical data were retrieved from `oci-registry.iad2.fedoraproject.org` machine logs.
The logs on the machine are kept only for few days. These data were obtained from 16th September 2023 to 21st September 2023 (it corresponds with release of Fedora 39 Beta).
These statistical data were retrieved from `oci-registry.iad2.fedoraproject.org` machine
logs. The logs on the machine are kept only for few days. These data were obtained from
16th September 2023 to 21st September 2023 (it corresponds with release of Fedora 39
Beta).
* Number of downloads per day: 800 - 1 000
* Number of requests per day: 350 000 - 400 000
- Number of downloads per day: 800 - 1 000
- Number of requests per day: 350 000 - 400 000
Investigation
-------------
The investigation is separated to multiple parts based on the current artifacts hosted on registry:
Fedora flatpaks, Fedora container images (base, minimal, coreos) and toolbox images.
The investigation is separated to multiple parts based on the current artifacts hosted
on registry: Fedora flatpaks, Fedora container images (base, minimal, coreos) and
toolbox images.
.. toctree::
:maxdepth: 1
@ -54,30 +62,32 @@ Fedora flatpaks, Fedora container images (base, minimal, coreos) and toolbox ima
toolbox
fedora_images
Conclusions
-----------
Migrating `registry.fedoraproject.org <https://registry.fedoraproject.org/>`_ to `quay.io <https://quay.io/>`_
doesn't seem to be initiative worthy. Especially with changes happening in Fedora 39.
There needs to be changes done to all three build pipelines we investigated, but those aren't complex.
Migrating `registry.fedoraproject.org <https://registry.fedoraproject.org/>`_ to
`quay.io <https://quay.io/>`_ doesn't seem to be initiative worthy. Especially with
changes happening in Fedora 39. There needs to be changes done to all three build
pipelines we investigated, but those aren't complex.
Recommendations
---------------
It will be best to first set everything on staging and then continue with production.
Only redirect the Fedora 39+ workflow and continue with step 4 after Fedora 38 will
be EOL.
Only redirect the Fedora 39+ workflow and continue with step 4 after Fedora 38 will be
EOL.
Proposed Roadmap
----------------
* Step 1 - Create namespaces corresponding to candidate-registry and registry on `quay.io <https://quay.io/>`_ (Optional: Create staging namespaces as well)
* Step 2 - Modify configurations and scripts (see corresponding investigation document for what needs
to be changed)
* Step 3 - Redirect proxies to `quay.io <https://quay.io/>`_
* Step 4 - Decommision `candidate-registry <https://candidate-registry.fedoraproject.org/>`_ and `candidate-registry <https://candidate-registry.fedoraproject.org/>`_
- Step 1 - Create namespaces corresponding to candidate-registry and registry on
`quay.io <https://quay.io/>`_ (Optional: Create staging namespaces as well)
- Step 2 - Modify configurations and scripts (see corresponding investigation document
for what needs to be changed)
- Step 3 - Redirect proxies to `quay.io <https://quay.io/>`_
- Step 4 - Decommision `candidate-registry
<https://candidate-registry.fedoraproject.org/>`_ and `candidate-registry
<https://candidate-registry.fedoraproject.org/>`_
Estimate of work
----------------

76
docs/registry_to_quay/toolbox.rst
View file

@ -3,22 +3,23 @@
Fedora Toolbx images
====================
This document contains information about current workflow used to build and publish Toolbox images.
It also describes changes needed to make it work with `quay.io <https://quay.io>`_.
This document contains information about current workflow used to build and publish
Toolbox images. It also describes changes needed to make it work with `quay.io
<https://quay.io>`_.
Current workflow
----------------
The toolbx images are currently built using OSBS. The OSBS submits builds to
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_ using
dockercfg and user can create an update in Bodhi. After testing period is finished or the update
gets enough karma it is moved by bodhi to
`registry.fedoraproject_org <https://registry.fedoraproject.org/>`_.
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_
using dockercfg and user can create an update in Bodhi. After testing period is finished
or the update gets enough karma it is moved by bodhi to `registry.fedoraproject_org
<https://registry.fedoraproject.org/>`_.
This will change in F39 where the images will be built by
`koji-flatpak <https://pagure.io/koji-flatpak>`_ plugin. Releng script will then submit the builds
to both `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and
`quay.io <https://quay.io>`_.
This will change in F39 where the images will be built by `koji-flatpak
<https://pagure.io/koji-flatpak>`_ plugin. Releng script will then submit the builds to
both `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and `quay.io
<https://quay.io>`_.
.. image:: ../_static/toolbox_workflow_registry.drawio.png
:target: ../_images/toolbox_workflow_registry.drawio.png
@ -31,36 +32,40 @@ work with current setup. It will go through pipeline step by step and describe w
needs to be changed in case of toolbx workflow.
OSBS
^^^^
~~~~
OSBS is using Openshift local container registry, which is configured by
`dockercfg <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for
`production <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_ and `staging <https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in
`setup-worker-namespace <https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_ playbook.
To make this work with `quay.io <https://quay.io>`_ we need to replace the dockercfg with new
one pointing to `quay.io <https://quay.io>`_.
OSBS is using Openshift local container registry, which is configured by `dockercfg
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
file and pointing to candidate-registry. There is dockercfg for `production
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-production-secret.j2>`_
and `staging
<https://pagure.io/fedora-infra/ansible/blob/main/f/files/osbs/dockercfg-staging-secret.j2>`_.
This dockercfg is used as a secret in OSBS and it's deployed in `setup-worker-namespace
<https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/osbs/setup-worker-namespace.yml>`_
playbook. To make this work with `quay.io <https://quay.io>`_ we need to replace the
dockercfg with new one pointing to `quay.io <https://quay.io>`_.
.. note::
OSBS could have trouble reaching to `quay.io <https://quay.io>`_, this needs to be solved for this to work.
OSBS could have trouble reaching to `quay.io <https://quay.io>`_, this needs to be
solved for this to work.
Releng script
^^^^^^^^^^^^^
~~~~~~~~~~~~~
In the new workflow introduced in Fedora 39+ is a Release Engineering script that pushes the image
to registry. This script could be found in
`releng repository <https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh>`_
and it's already pushing images to `quay.io <https://quay.io>`_.
In the new workflow introduced in Fedora 39+ is a Release Engineering script that pushes
the image to registry. This script could be found in `releng repository
<https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh>`_ and
it's already pushing images to `quay.io <https://quay.io>`_.
Bodhi
^^^^^
~~~~~
Bodhi is pushing the container builds from
`candidate-registry.fedoraproject.org <https://candidate-registry.fedoraproject.org/>`_ to
`registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ and this is configured in
`production.ini template <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
Bodhi is pushing the container builds from `candidate-registry.fedoraproject.org
<https://candidate-registry.fedoraproject.org/>`_ to `registry.fedoraproject_org
<https://registry.fedoraproject.org/>`_ and this is configured in `production.ini
template
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/base/templates/production.ini.j2>`_
in ansible.
To make it work with quay.io we just need to change `container.destination_registry` and
@ -68,11 +73,12 @@ To make it work with quay.io we just need to change `container.destination_regis
.. note::
Bodhi wouldn't be part of the toolbx workflow in Fedora 39 and later. So no need to change
anything here.
Bodhi wouldn't be part of the toolbx workflow in Fedora 39 and later. So no need to
change anything here.
Toolbx
^^^^^^
~~~~~~
Toolbx has the `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ set in code.
This needs to be changed before we completely switch to `quay.io <https://quay.io>`_.
Toolbx has the `registry.fedoraproject_org <https://registry.fedoraproject.org/>`_ set
in code. This needs to be changed before we completely switch to `quay.io
<https://quay.io>`_.

76
docs/webhook2fedmsg/design_of_webhook2fedmsg.rst
View file

@ -6,92 +6,98 @@ Design document for webhook2fedmsg
This document will investigate the possibility to create a more general solution
converting various webhooks to fedora messaging messages.
External services
-----------------
External services webhook2fedmsg will communicate with.
* Database
- Database
Will be used to store data about FAS user and tokens they added. We can consider supporting multiple tokens for each service per user.
For database it would be best to use existing `PostreSQL <https://pagure.io/fedora-infra/ansible/blob/main/f/roles/postgresql_server>`_
Will be used to store data about FAS user and tokens they added. We can consider
supporting multiple tokens for each service per user. For database it would be best to
use existing `PostreSQL
<https://pagure.io/fedora-infra/ansible/blob/main/f/roles/postgresql_server>`_
instance we already host in Fedora infra.
* Webhook services
- Webhook services
The application will listen for webhooks from supported services. For each service should be separate backend, so it's easier to
remove/add new services.
The application will listen for webhooks from supported services. For each service
should be separate backend, so it's easier to remove/add new services.
* FAS
We need to authenticate user against Fedora Account System.
For authentication we can use `flask-oidc <https://github.com/fedora-infra/flask-oidc>`_ or `authlib <https://docs.authlib.org/en/latest/>`_
if `Flask <https://flask.palletsprojects.com/en/2.2.x/>`_ is not a framework of choice.
- FAS
We need to authenticate user against Fedora Account System. For authentication we can
use `flask-oidc <https://github.com/fedora-infra/flask-oidc>`_ or `authlib
<https://docs.authlib.org/en/latest/>`_ if `Flask
<https://flask.palletsprojects.com/en/2.2.x/>`_ is not a framework of choice.
Use cases
---------
Use cases that need to be implemented by webhook2fedmsg.
* User login
- User login
User will be able to login through web interface using FAS account.
* User can add token for service
- User can add token for service
When logged in user can add token for any service known by webhook2fedmsg. We should support multiple tokens per service.
When logged in user can add token for any service known by webhook2fedmsg. We should
support multiple tokens per service.
* User can modify/remove existing tokens
- User can modify/remove existing tokens
When logged in user should be able to modify/remove tokens for services he added earlier.
When logged in user should be able to modify/remove tokens for services he added
earlier.
* User can list already added tokens
- User can list already added tokens
When logged in user should be able to see tokens he already added and for which service they are.
When logged in user should be able to see tokens he already added and for which
service they are.
* User can see the URL for webhook
- User can see the URL for webhook
webhook2fedmsg should show on web interface URL that could be used to set up webhook on the service.
webhook2fedmsg should show on web interface URL that could be used to set up webhook
on the service.
* Listening for webhook POST requests
- Listening for webhook POST requests
The application will provide endpoint for incoming requests. Ideally one per service.
The webhook needs to be set by user on webhook service page.
* Validating webhook POST requests
- Validating webhook POST requests
Every incoming request from webhook services will be validated against the token database and search if it is valid token for service.
It will drop any invalid request.
Every incoming request from webhook services will be validated against the token
database and search if it is valid token for service. It will drop any invalid
request.
* Processing webhook POST requests
Each message needs to be converted from webhook JSON to Fedora message. The webhook2fedmsg should provide Fedora messaging schema for topics emitted by it.
- Processing webhook POST requests
Each message needs to be converted from webhook JSON to Fedora message. The
webhook2fedmsg should provide Fedora messaging schema for topics emitted by it.
Endpoints
---------
Endpoints that should be provided by webhook2fedmsg.
* `/`
- `/`
Index page. User is able to login here.
* `/tokens`
- `/tokens`
Token management page shows most information relevant to user - URLs for webhook endpoints, management of tokens.
Token management page shows most information relevant to user - URLs for webhook
endpoints, management of tokens.
* `/login`
- `/login`
Login page user is automatically redirected to, if not authenticated.
* `/<service>`
- `/<service>`
Endpoint where webhook service should sent POST requests.
* `/api`
- `/api`
Endpoint for API requests. API will be a nice to have alternative to web GUI.

87
docs/webhook2fedmsg/index.rst
View file

@ -1,50 +1,52 @@
webhook2fedmsg
=============
==============
Purpose
-------
This investigation's goal is to explore a possible new application that will
convert webhooks from different services to Fedora messaging messages.
This investigation's goal is to explore a possible new application that will convert
webhooks from different services to Fedora messaging messages.
Resources
---------
* Initiative proposal: https://pagure.io/cpe/initiatives-proposal/issue/18
- Initiative proposal: https://pagure.io/cpe/initiatives-proposal/issue/18
Requirements
------------
* User can log in using FAS account
* User can link different webhook services to FAS account
* Application will listen for the webhook POST requests
* Application will provide endpoint for each service
* POST request will be validated against the known tokens for the service
* Support for `GitHub <https://docs.github.com/en/developers/webhooks-and-events/webhooks/creating-webhooks>`_
* Support for `Discourse <https://meta.discourse.org/t/configure-webhooks-that-trigger-on-discourse-events-to-integrate-with-external-services/49045>`_
* CI - Recommending `Zuul CI <https://fedoraproject.org/wiki/Zuul-based-ci>`_
or `Github Actions <https://github.com/features/actions>`_
* Documentation - Documentation could be hosted on `readthedocs.org <https://readthedocs.org/>`_
* OpenShift deployment
* Unit tests - Recommending `tox <https://tox.wiki/en/latest/>`_
* Fedora messaging schema
* Easy to use dev env - Recommending either `vagrant <https://www.vagrantup.com/>`_
or containerized environment
- User can log in using FAS account
- User can link different webhook services to FAS account
- Application will listen for the webhook POST requests
- Application will provide endpoint for each service
- POST request will be validated against the known tokens for the service
- Support for `GitHub
<https://docs.github.com/en/developers/webhooks-and-events/webhooks/creating-webhooks>`_
- Support for `Discourse
<https://meta.discourse.org/t/configure-webhooks-that-trigger-on-discourse-events-to-integrate-with-external-services/49045>`_
- CI - Recommending `Zuul CI <https://fedoraproject.org/wiki/Zuul-based-ci>`_ or `Github
Actions <https://github.com/features/actions>`_
- Documentation - Documentation could be hosted on `readthedocs.org
<https://readthedocs.org/>`_
- OpenShift deployment
- Unit tests - Recommending `tox <https://tox.wiki/en/latest/>`_
- Fedora messaging schema
- Easy to use dev env - Recommending either `vagrant <https://www.vagrantup.com/>`_ or
containerized environment
Nice to have
------------
List of features that would be nice to have.
* Document how to create webhooks on every service
* Document how to generate token on every service
* Support for `GitLab <http://mmb.irbbarcelona.org/gitlab/help/web_hooks/web_hooks.md>`_
* Support for `MediaWiki <https://www.mediawiki.org/wiki/Extension:EventBus>`_
* Automatic dependency management - Recommending `renovate <https://docs.renovatebot.com/>`_
* API for managing tokens
* Automatic deployment in OpenShift
- Document how to create webhooks on every service
- Document how to generate token on every service
- Support for `GitLab <http://mmb.irbbarcelona.org/gitlab/help/web_hooks/web_hooks.md>`_
- Support for `MediaWiki <https://www.mediawiki.org/wiki/Extension:EventBus>`_
- Automatic dependency management - Recommending `renovate
<https://docs.renovatebot.com/>`_
- API for managing tokens
- Automatic deployment in OpenShift
Investigation
-------------
@ -56,7 +58,6 @@ Following it the design document for webhook2fedmsg.
design_of_webhook2fedmsg
The Good Points
---------------
@ -66,35 +67,33 @@ The Good Points
4. Users can manage tokens by themselves
5. With this solution we don't need to rewrite github2fedmsg - one less initiative
The Bad points
--------------
1. Initial work - completely new app
2. Another application to maintain - However this will allow us to get rid of few others
2. Manual work with tokens and webhooks on webhook service side for users
3. Manual work with tokens and webhooks on webhook service side for users
Conclusions
-----------
This application will help us to lower our codebase. We wouldn't need to maintain multiple
applications service2fedmsg anymore, but everything that provides webhook could
be managed by this application.
It is definitely something we have knowledge and skills to do.
This application will help us to lower our codebase. We wouldn't need to maintain
multiple applications service2fedmsg anymore, but everything that provides webhook could
be managed by this application. It is definitely something we have knowledge and skills
to do.
Proposed Roadmap
----------------
* Step 1 - Write new app with tests and documentation
* Step 2 - Create separate repository with message schemas
* Step 3 - Deploy in staging
* Step 4 - Inform community this service is available for testing
* Step 5 - Deploy in production
* Step 6 - Community blog announcing this new service
- Step 1 - Write new app with tests and documentation
- Step 2 - Create separate repository with message schemas
- Step 3 - Deploy in staging
- Step 4 - Inform community this service is available for testing
- Step 5 - Deploy in production
- Step 6 - Community blog announcing this new service
Estimate of work
----------------
This work will need at least 2 developers, one with web frontend skill set.
The estimation for this project is 12 weeks.
This work will need at least 2 developers, one with web frontend skill set. The
estimation for this project is 12 weeks.

32
docs/workflow.rst
View file

@ -1,31 +1,29 @@
Workflow
========
The ARC teams does the investigation work on the
`initiatives <https://docs.fedoraproject.org/en-US/cpe/initiatives/>`_ proposed
to the CPE team.
The ARC teams does the investigation work on the `initiatives
<https://docs.fedoraproject.org/en-US/cpe/initiatives/>`_ proposed to the CPE team.
Here belows is a schemas explaining its workflow.
.. image:: _static/ARC_Workflow.jpg
:target: _images/ARC_Workflow.jpg
:target: _images/ARC_Workflow.jpg
The workflow can be summarized as follow:
* Initiatives are submitted to CPE's project owner who will review them.
* If the initiatives are deemed in scope for the CPE team, they will be accepted.
* The CPE team will prioritize the initiatives against each other, leading to a
- Initiatives are submitted to CPE's project owner who will review them.
- If the initiatives are deemed in scope for the CPE team, they will be accepted.
- The CPE team will prioritize the initiatives against each other, leading to a
prioritized backlog of initiatives.
* The ARC team will take the backlog in that order and for each initiative it will:
- The ARC team will take the backlog in that order and for each initiative it will:
* Reach out to the CPE team and its communities (as needed) for input on potential
- Reach out to the CPE team and its communities (as needed) for input on potential
solution to implement the initiative
* Do the investigation/prototyping
* Report to the CPE team and its communities (as needed) its finding together
with a recommendation
- Do the investigation/prototyping
- Report to the CPE team and its communities (as needed) its finding together with a
recommendation
* The initiative will then be scheduled for the next prioritization meeting with
its stakeholders.
* When the initiative is scheduled, the CPE team will take it and implement it
using the information, experiment and prototypes gathered and built by the ARC
team.
- The initiative will then be scheduled for the next prioritization meeting with its
stakeholders.
- When the initiative is scheduled, the CPE team will take it and implement it using the
information, experiment and prototypes gathered and built by the ARC team.