fix parsing errors and sphinx warnings
Signed-off-by: Ryan Lerch <rlerch@redhat.com>
This commit is contained in:
parent
8fb9b2fdf0
commit
ba720c3d77
98 changed files with 4799 additions and 4788 deletions
|
@ -1,170 +1,166 @@
|
|||
.. _enhancing_the_development_environment:
|
||||
|
||||
Enhancing the Development Environment
|
||||
========================================
|
||||
=====================================
|
||||
|
||||
The current state of the development environment of the github2fedmsg project
|
||||
leaves a lot to be desired. This document attempts to address all the existing
|
||||
tech debts bound to the github2fedmsg project repository.
|
||||
The current state of the development environment of the github2fedmsg project leaves a
|
||||
lot to be desired. This document attempts to address all the existing tech debts bound
|
||||
to the github2fedmsg project repository.
|
||||
|
||||
Investigations
|
||||
----
|
||||
--------------
|
||||
|
||||
**Continuous Integration**
|
||||
1. Apparently, there has been a continuous integration configured for the
|
||||
repository, which was last active on Jun 17, 2020 on
|
||||
`PR #27 <https://github.com/fedora-infra/github2fedmsg/pull/27>`_.
|
||||
This continuous integration was set up on a Jenkins instance of Fedora
|
||||
Infrastructure on the CentOS CI namespace (as deduced by the `details
|
||||
URL <https://jenkins-fedora-apps.apps.ci.centos.org/job/github2fedmsg/job/PR-27/1/display/redirect>`_)
|
||||
but is either no longer available or can only be accessed via the Red Hat
|
||||
VPN. If the former is the case, we would want to reconfigure the
|
||||
continuous integration in the project repository but if latter is the
|
||||
case, we would want to ensure that the continuous integration is
|
||||
accessible to the wider community to promote contributor participation in
|
||||
the project.
|
||||
2. In order to set up a continuous integration in the project repository,
|
||||
`GitHub Actions <https://github.com/features/actions>`_ can be put into
|
||||
use as the project repository itself is hosted on GitHub, thus allowing
|
||||
for integration and support to a greater extent. It is also possible to
|
||||
use Fedora Linux containers in GitHub Actions, as evidenced by their use
|
||||
in `Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/.github/workflows/main.yml#L10>`_,
|
||||
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/.github/workflows/main.yaml#L8>`_,
|
||||
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/.github/workflows/main.yaml#L8>`_,
|
||||
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/.github/workflows/main.yml#L9>`_
|
||||
and many more such projects, which would help replicate the production
|
||||
environment accurately for the relevant build and test jobs. Other
|
||||
continuous integration options like `Zuul CI <https://zuul-ci.org/>`_,
|
||||
`Travis CI <https://www.travis-ci.com/>`_ etc. can also be considered on
|
||||
the basis of particular needs.
|
||||
1. Apparently, there has been a continuous integration configured for the
|
||||
repository, which was last active on Jun 17, 2020 on `PR #27
|
||||
<https://github.com/fedora-infra/github2fedmsg/pull/27>`_. This continuous
|
||||
integration was set up on a Jenkins instance of Fedora Infrastructure on the
|
||||
CentOS CI namespace (as deduced by the `details URL
|
||||
<https://jenkins-fedora-apps.apps.ci.centos.org/job/github2fedmsg/job/PR-27/1/display/redirect>`_)
|
||||
but is either no longer available or can only be accessed via the Red Hat VPN. If
|
||||
the former is the case, we would want to reconfigure the continuous integration
|
||||
in the project repository but if latter is the case, we would want to ensure that
|
||||
the continuous integration is accessible to the wider community to promote
|
||||
contributor participation in the project.
|
||||
2. In order to set up a continuous integration in the project repository, `GitHub
|
||||
Actions <https://github.com/features/actions>`_ can be put into use as the
|
||||
project repository itself is hosted on GitHub, thus allowing for integration and
|
||||
support to a greater extent. It is also possible to use Fedora Linux containers
|
||||
in GitHub Actions, as evidenced by their use in `Meetbot Logs
|
||||
<https://github.com/fedora-infra/mote/blob/main/.github/workflows/main.yml#L10>`__,
|
||||
`CentOS Duffy
|
||||
<https://github.com/CentOS/duffy/blob/dev/.github/workflows/main.yaml#L8>`_,
|
||||
`Fedora Messaging Notifier
|
||||
<https://github.com/fedora-infra/fmn/blob/fmn-next/.github/workflows/main.yaml#L8>`__,
|
||||
`MDAPI
|
||||
<https://github.com/fedora-infra/mdapi/blob/develop/.github/workflows/main.yml#L9>`_
|
||||
and many more such projects, which would help replicate the production
|
||||
environment accurately for the relevant build and test jobs. Other continuous
|
||||
integration options like `Zuul CI <https://zuul-ci.org/>`_, `Travis CI
|
||||
<https://www.travis-ci.com/>`_ etc. can also be considered on the basis of
|
||||
particular needs.
|
||||
|
||||
**Development Environment**
|
||||
1. A small amount of information about setting up a virtual environment and
|
||||
installing the project using
|
||||
`setuptools <https://setuptools.pypa.io/en/latest/>`_
|
||||
is currently available on the
|
||||
`README.rst document <https://github.com/fedora-infra/github2fedmsg/blob/develop/README.rst>`_
|
||||
in the project repository. Although, the current setup works fine in a
|
||||
specific environment having (now EOLed) Python 2.7.x, it can be useful to
|
||||
abstract the use of setuptools, using dependency management tools like
|
||||
`poetry <https://python-poetry.org/>`_,
|
||||
`pipenv <https://pipenv.pypa.io/en/latest/>`_ etc.
|
||||
to steer away from the unnecessary attention in creating and updating
|
||||
the setuptools configuration, facilitating for automatic dependency
|
||||
updates and maintaining a list of dependencies specific to development,
|
||||
testing and production scenarios.
|
||||
2. Maintaining codebase to uphold established quality standards will be of
|
||||
great help, both, for the convenience of existing contributors and to
|
||||
help onboard potential ones. In the current setup,
|
||||
`flake8 <https://flake8.pycqa.org/en/latest/>`_ is
|
||||
`used <https://github.com/fedora-infra/github2fedmsg/blob/develop/tox.ini#L13>`_
|
||||
for linting the codebase. In addition to flake8,
|
||||
`isort <https://isort.readthedocs.io/en/latest/>`_ and
|
||||
`black <https://black.readthedocs.io/en/stable/>`_ can
|
||||
be used for sorting of dependency imports and enforcing code styles
|
||||
wherever necessary. This has been of great help in maintaining code
|
||||
quality, as evidenced by their uses in
|
||||
`Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/tox.ini#L27-L56>`_,
|
||||
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/tox.ini#L25-L41>`_,
|
||||
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L16-L33>`_,
|
||||
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L26-L54>`_
|
||||
and many more such projects.
|
||||
1. A small amount of information about setting up a virtual environment and
|
||||
installing the project using `setuptools
|
||||
<https://setuptools.pypa.io/en/latest/>`_ is currently available on the
|
||||
`README.rst document
|
||||
<https://github.com/fedora-infra/github2fedmsg/blob/develop/README.rst>`_ in the
|
||||
project repository. Although, the current setup works fine in a specific
|
||||
environment having (now EOLed) Python 2.7.x, it can be useful to abstract the use
|
||||
of setuptools, using dependency management tools like `poetry
|
||||
<https://python-poetry.org/>`_, `pipenv <https://pipenv.pypa.io/en/latest/>`_
|
||||
etc. to steer away from the unnecessary attention in creating and updating the
|
||||
setuptools configuration, facilitating for automatic dependency updates and
|
||||
maintaining a list of dependencies specific to development, testing and
|
||||
production scenarios.
|
||||
2. Maintaining codebase to uphold established quality standards will be of great
|
||||
help, both, for the convenience of existing contributors and to help onboard
|
||||
potential ones. In the current setup, `flake8
|
||||
<https://flake8.pycqa.org/en/latest/>`__ is `used
|
||||
<https://github.com/fedora-infra/github2fedmsg/blob/develop/tox.ini#L13>`__ for
|
||||
linting the codebase. In addition to flake8, `isort
|
||||
<https://isort.readthedocs.io/en/latest/>`__ and `black
|
||||
<https://black.readthedocs.io/en/stable/>`__ can be used for sorting of
|
||||
dependency imports and enforcing code styles wherever necessary. This has been of
|
||||
great help in maintaining code quality, as evidenced by their uses in `Meetbot
|
||||
Logs <https://github.com/fedora-infra/mote/blob/main/tox.ini#L27-L56>`__, `CentOS
|
||||
Duffy <https://github.com/CentOS/duffy/blob/dev/tox.ini#L25-L41>`__, `Fedora
|
||||
Messaging Notifier
|
||||
<https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L16-L33>`__, `MDAPI
|
||||
<https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L26-L54>`__ and many
|
||||
more such projects.
|
||||
|
||||
**Automatic Dependency Updates**
|
||||
1. Projects such as
|
||||
`Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/.github/dependabot.yml>`_,
|
||||
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/.github/dependabot.yml>`_ and
|
||||
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/.github/dependabot.yml>`_
|
||||
use `GitHub Dependabot <https://github.com/dependabot>`_ to help
|
||||
automatically update the dependency lockfile as and when new updates and
|
||||
vulnerability fixes come out for the unexcluded dependencies, matching
|
||||
the version requirement mentioned in the project configuration file.
|
||||
While the former case requires having a valid GitHub Dependabot
|
||||
configuration in the repository, the latter does not and will work
|
||||
regardless of having it or not. Being GitHub's own security worker bot,
|
||||
it has a greater extent of integration with the project repositories
|
||||
available on GitHub.
|
||||
2. Projects such as
|
||||
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/renovate.json>`_
|
||||
and `Anitya <https://github.com/fedora-infra/anitya/blob/master/renovate.json>`_
|
||||
use `Renovate <https://github.com/apps/renovate>`_ to help automatically
|
||||
update the dependency lockfile as and when new updates or vulnerability
|
||||
fixes come out for the unexcluded dependencies, matching the version
|
||||
requirement mentioned in the project configuration file. Renovate is
|
||||
highly flexible with its configuration, allows for checking of dependency
|
||||
lockfile for verifying checksums, organizing custom schedules for updates
|
||||
and also supports auto-merging of the dependency update pull requests.
|
||||
1. Projects such as `Meetbot Logs
|
||||
<https://github.com/fedora-infra/mote/blob/main/.github/dependabot.yml>`__,
|
||||
`CentOS Duffy
|
||||
<https://github.com/CentOS/duffy/blob/dev/.github/dependabot.yml>`__ and `MDAPI
|
||||
<https://github.com/fedora-infra/mdapi/blob/develop/.github/dependabot.yml>`__
|
||||
use `GitHub Dependabot <https://github.com/dependabot>`__ to help automatically
|
||||
update the dependency lockfile as and when new updates and vulnerability fixes
|
||||
come out for the unexcluded dependencies, matching the version requirement
|
||||
mentioned in the project configuration file. While the former case requires
|
||||
having a valid GitHub Dependabot configuration in the repository, the latter does
|
||||
not and will work regardless of having it or not. Being GitHub's own security
|
||||
worker bot, it has a greater extent of integration with the project repositories
|
||||
available on GitHub.
|
||||
2. Projects such as `Fedora Messaging Notifier
|
||||
<https://github.com/fedora-infra/fmn/blob/fmn-next/renovate.json>`__ and `Anitya
|
||||
<https://github.com/fedora-infra/anitya/blob/master/renovate.json>`_ use
|
||||
`Renovate <https://github.com/apps/renovate>`_ to help automatically update the
|
||||
dependency lockfile as and when new updates or vulnerability fixes come out for
|
||||
the unexcluded dependencies, matching the version requirement mentioned in the
|
||||
project configuration file. Renovate is highly flexible with its configuration,
|
||||
allows for checking of dependency lockfile for verifying checksums, organizing
|
||||
custom schedules for updates and also supports auto-merging of the dependency
|
||||
update pull requests.
|
||||
|
||||
**Outdated dependencies**
|
||||
1. The project makes use of a certain set of dependencies which are either
|
||||
left in a pre-release state or are no longer actively maintained. While
|
||||
these dependencies can be put to use in the project in its current state,
|
||||
it is very likely that a lot of security vulnerabilities must have flown
|
||||
under the radar unpatched, in the absence of features updates, security
|
||||
patches and bug fixes. Apart from that, being older libraries/frameworks -
|
||||
the availability of updated documentation is likely to be scarce for
|
||||
these, thus making the development using these dependencies more
|
||||
challenging for the maintainers.
|
||||
2. Moving forward, we would want to look into either possible replacing the
|
||||
following dependencies or avoiding their use entirely.
|
||||
1. The project makes use of a certain set of dependencies which are either left in a
|
||||
pre-release state or are no longer actively maintained. While these dependencies
|
||||
can be put to use in the project in its current state, it is very likely that a
|
||||
lot of security vulnerabilities must have flown under the radar unpatched, in the
|
||||
absence of features updates, security patches and bug fixes. Apart from that,
|
||||
being older libraries/frameworks - the availability of updated documentation is
|
||||
likely to be scarce for these, thus making the development using these
|
||||
dependencies more challenging for the maintainers.
|
||||
2. Moving forward, we would want to look into either possible replacing the
|
||||
following dependencies or avoiding their use entirely.
|
||||
|
||||
1. `Velruse <https://pypi.org/project/velruse/>`_, had its last
|
||||
update almost *9 years back* in
|
||||
`Aug 30, 2013 <https://pypi.org/project/velruse/1.1.1/>`_.
|
||||
2. `WebError <https://pypi.org/project/WebError/>`_, had its last update
|
||||
almost *6 years back* in
|
||||
`Apr 10, 2016 <https://pypi.org/project/WebError/0.13.1/>`_ and the
|
||||
`project documentation states <https://github.com/Pylons/weberror/blob/e395f6d0f0becb299b905197c4577d3b74ad40ae/README.rst#notice>`_
|
||||
that the project is no longer actively maintained.
|
||||
3. `pyramid-mako <https://pypi.org/project/pyramid-mako/>`_ had its last
|
||||
update almost *3 years back* in
|
||||
`Aug 19, 2019 <https://pypi.org/project/pyramid-mako/1.1.0/>`_.
|
||||
4. `transaction <https://pypi.org/project/transaction/>`_ had its last
|
||||
update almost *2 years back* in
|
||||
`Dec 11, 2020 <https://pypi.org/project/transaction/3.0.1/>`_.
|
||||
1. `Velruse <https://pypi.org/project/velruse/>`_, had its last update almost *9
|
||||
years back* in `Aug 30, 2013 <https://pypi.org/project/velruse/1.1.1/>`_.
|
||||
2. `WebError <https://pypi.org/project/WebError/>`_, had its last update almost
|
||||
*6 years back* in `Apr 10, 2016 <https://pypi.org/project/WebError/0.13.1/>`_
|
||||
and the `project documentation states
|
||||
<https://github.com/Pylons/weberror/blob/e395f6d0f0becb299b905197c4577d3b74ad40ae/README.rst#notice>`_
|
||||
that the project is no longer actively maintained.
|
||||
3. `pyramid-mako <https://pypi.org/project/pyramid-mako/>`_ had its last update
|
||||
almost *3 years back* in `Aug 19, 2019
|
||||
<https://pypi.org/project/pyramid-mako/1.1.0/>`_.
|
||||
4. `transaction <https://pypi.org/project/transaction/>`_ had its last update
|
||||
almost *2 years back* in `Dec 11, 2020
|
||||
<https://pypi.org/project/transaction/3.0.1/>`_.
|
||||
|
||||
**Contributor's Quality of Life**
|
||||
1. Certain additional documentations such as CODE OF CONDUCT, CONTRIBUTING
|
||||
GUIDELINES etc. can either be added or linked in, to set the expectations
|
||||
right for both existing and potential contributors. The wiki feature
|
||||
present in the project repository can be preferred to be put into use
|
||||
for keeping documentation like these which are less likely to change
|
||||
across multiple releases. These documentation are available in
|
||||
repositories such as MDAPI
|
||||
`[1] <https://github.com/fedora-infra/mdapi/blob/develop/CODE_OF_CONDUCT.md>`_
|
||||
`[2] <https://github.com/fedora-infra/mdapi/blob/develop/CONTRIBUTING.md>`_,
|
||||
Meetbot Logs
|
||||
`[1] <https://github.com/fedora-infra/mote/blob/main/CODE_OF_CONDUCT.md>`_
|
||||
`[2] <https://github.com/fedora-infra/mote/blob/main/CONTRIBUTING.md>`_,
|
||||
CentOS Duffy
|
||||
`[1] <https://github.com/CentOS/duffy/blob/dev/CONTRIBUTING.md>`_ etc.
|
||||
and can be included here too.
|
||||
2. Tests and code coverage should be added to ensure that the project code
|
||||
remains functional over time across the addition of features, fixing of
|
||||
bugs, patching of security vulnerabilities etc. The
|
||||
`tests <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/tests.py>`_
|
||||
currently available in the project repository have a limited scope and
|
||||
are not automated using continuous integration to test out the code
|
||||
`on pull requests <https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#pull_request>`_,
|
||||
`on pushes <https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#push>`_
|
||||
and
|
||||
`before deployment workflows <https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#workflow_dispatch>`_.
|
||||
These tests and code coverage have been implemented in
|
||||
`Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/tox.ini#L7-L19>`_,
|
||||
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/tox.ini#L7-L18>`_,
|
||||
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L7-L14>`_,
|
||||
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L7-L17>`_
|
||||
and many more such repositories and are automated using GitHub Actions.
|
||||
1. Certain additional documentations such as CODE OF CONDUCT, CONTRIBUTING
|
||||
GUIDELINES etc. can either be added or linked in, to set the expectations right
|
||||
for both existing and potential contributors. The wiki feature present in the
|
||||
project repository can be preferred to be put into use for keeping documentation
|
||||
like these which are less likely to change across multiple releases. These
|
||||
documentation are available in repositories such as MDAPI `[1]
|
||||
<https://github.com/fedora-infra/mdapi/blob/develop/CODE_OF_CONDUCT.md>`__ `[2]
|
||||
<https://github.com/fedora-infra/mdapi/blob/develop/CONTRIBUTING.md>`__, Meetbot
|
||||
Logs `[1] <https://github.com/fedora-infra/mote/blob/main/CODE_OF_CONDUCT.md>`__
|
||||
`[2] <https://github.com/fedora-infra/mote/blob/main/CONTRIBUTING.md>`__, CentOS
|
||||
Duffy `[1] <https://github.com/CentOS/duffy/blob/dev/CONTRIBUTING.md>`__ etc. and
|
||||
can be included here too.
|
||||
2. Tests and code coverage should be added to ensure that the project code remains
|
||||
functional over time across the addition of features, fixing of bugs, patching of
|
||||
security vulnerabilities etc. The `tests
|
||||
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/tests.py>`_
|
||||
currently available in the project repository have a limited scope and are not
|
||||
automated using continuous integration to test out the code `on pull requests
|
||||
<https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#pull_request>`_,
|
||||
`on pushes
|
||||
<https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#push>`_
|
||||
and `before deployment workflows
|
||||
<https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#workflow_dispatch>`_.
|
||||
These tests and code coverage have been implemented in `Meetbot Logs
|
||||
<https://github.com/fedora-infra/mote/blob/main/tox.ini#L7-L19>`__, `CentOS Duffy
|
||||
<https://github.com/CentOS/duffy/blob/dev/tox.ini#L7-L18>`__, `Fedora Messaging
|
||||
Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L7-L14>`__,
|
||||
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L7-L17>`__ and
|
||||
many more such repositories and are automated using GitHub Actions.
|
||||
|
||||
**Automating Deployments to OpenShift**
|
||||
1. The project currently makes use of semantic versioning for its releases
|
||||
and requires manual intervention for deploying every distinct release to
|
||||
both `staging <https://apps.fedoraproject.org/github2fedmsg>`_ and
|
||||
`production <https://apps.stg.fedoraproject.org/github2fedmsg>`_
|
||||
environments. The effort can be reduced by the use of webhooks which
|
||||
would help monitor distinct release on the project repository and then
|
||||
automatically run deployment scripts to put the new version up on
|
||||
the staging environment. For this, GitHub webhooks can be configured to
|
||||
be used on the said OpenShift cluster and the production deployment is
|
||||
suggested to require manual intervention to make way for additional due
|
||||
diligence before deployment.
|
||||
1. The project currently makes use of semantic versioning for its releases and
|
||||
requires manual intervention for deploying every distinct release to both
|
||||
`staging <https://apps.fedoraproject.org/github2fedmsg>`_ and `production
|
||||
<https://apps.stg.fedoraproject.org/github2fedmsg>`_ environments. The effort can
|
||||
be reduced by the use of webhooks which would help monitor distinct release on
|
||||
the project repository and then automatically run deployment scripts to put the
|
||||
new version up on the staging environment. For this, GitHub webhooks can be
|
||||
configured to be used on the said OpenShift cluster and the production deployment
|
||||
is suggested to require manual intervention to make way for additional due
|
||||
diligence before deployment.
|
||||
|
|
|
@ -5,10 +5,10 @@ Purpose
|
|||
-------
|
||||
|
||||
This investigation's goal is to address tech debt of github2fedmsg application.
|
||||
github2fedmsg application is used to translate events emitted by github to fedora messaging
|
||||
messages using github webhooks. It also has database of projects to watch and provides web
|
||||
interface to add new repositories. It authenticates with FAS and Github, linking accounts
|
||||
together.
|
||||
github2fedmsg application is used to translate events emitted by github to fedora
|
||||
messaging messages using github webhooks. It also has database of projects to watch and
|
||||
provides web interface to add new repositories. It authenticates with FAS and Github,
|
||||
linking accounts together.
|
||||
|
||||
The app has git-hub2fedmsg bot on GitHub that is being used to link the FAS account to
|
||||
Github.
|
||||
|
@ -16,34 +16,35 @@ Github.
|
|||
Resources
|
||||
---------
|
||||
|
||||
* github2fedmsg initiative proposal: https://pagure.io/cpe/initiatives-proposal/issue/24
|
||||
* github2fedmsg repo: https://github.com/fedora-infra/github2fedmsg
|
||||
* github2fedmsg web: https://apps.fedoraproject.org/github2fedmsg
|
||||
* github2fedmsg ansible: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/github2fedmsg
|
||||
* github2fedmsg SOP: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/github2fedmsg/
|
||||
- github2fedmsg initiative proposal: https://pagure.io/cpe/initiatives-proposal/issue/24
|
||||
- github2fedmsg repo: https://github.com/fedora-infra/github2fedmsg
|
||||
- github2fedmsg web: https://apps.fedoraproject.org/github2fedmsg
|
||||
- github2fedmsg ansible:
|
||||
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/github2fedmsg
|
||||
- github2fedmsg SOP:
|
||||
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/github2fedmsg/
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
* Porting to Flask and authlib
|
||||
* Linking GitHub account to FAS account
|
||||
* Listening to GitHub events
|
||||
* Translating GitHub events to Fedora messages
|
||||
* Add/remove repositories to watch
|
||||
* CI
|
||||
* Documentation
|
||||
* OpenShift deployment
|
||||
|
||||
- Porting to Flask and authlib
|
||||
- Linking GitHub account to FAS account
|
||||
- Listening to GitHub events
|
||||
- Translating GitHub events to Fedora messages
|
||||
- Add/remove repositories to watch
|
||||
- CI
|
||||
- Documentation
|
||||
- OpenShift deployment
|
||||
|
||||
What Needs To Be Addressed?
|
||||
---------------------------
|
||||
|
||||
* Create message schema
|
||||
* python2 -> python3
|
||||
* migration to OpenShift
|
||||
* Documentation - only README and SOPs are available
|
||||
* Update development environment
|
||||
* There is no staging environment
|
||||
- Create message schema
|
||||
- python2 -> python3
|
||||
- migration to OpenShift
|
||||
- Documentation - only README and SOPs are available
|
||||
- Update development environment
|
||||
- There is no staging environment
|
||||
|
||||
Investigation
|
||||
-------------
|
||||
|
@ -57,12 +58,11 @@ Here is the list of ideas/things we discussed/looked at:
|
|||
rewrite
|
||||
enhancing_the_development_environment
|
||||
|
||||
|
||||
Conclusions
|
||||
-----------
|
||||
|
||||
Investigation team recommends to not work on this app, but instead work
|
||||
on :doc:`../webhook2fedmsg/index`.
|
||||
Investigation team recommends to not work on this app, but instead work on
|
||||
:doc:`../webhook2fedmsg/index`.
|
||||
|
||||
Proposed Roadmap
|
||||
----------------
|
||||
|
|
|
@ -3,57 +3,71 @@
|
|||
Revitalization of github2fedmsg application
|
||||
===========================================
|
||||
|
||||
This document will investigate the possibility of updating the code base we currently have.
|
||||
This document will investigate the possibility of updating the code base we currently
|
||||
have.
|
||||
|
||||
What needs to be addressed (from our index)
|
||||
-------------------------------------------
|
||||
|
||||
**Python 2 -> Python 3**
|
||||
| *Current situation*: Written in Python 2.7.x which is now `EOLed <https://www.python.org/doc/sunset-python-2/>`_.
|
||||
| *Suggested outcome*: Python 3.10.x (and above).
|
||||
| *Reason(s)*: Benefits from performance improvements, bug fixes, security patches etc. of the language.
|
||||
| *Current situation*: Written in Python 2.7.x which is now `EOLed
|
||||
<https://www.python.org/doc/sunset-python-2/>`_.
|
||||
| *Suggested outcome*: Python 3.10.x (and above).
|
||||
| *Reason(s)*: Benefits from performance improvements, bug fixes, security patches
|
||||
etc. of the language.
|
||||
|
||||
**Migration to OpenShift**
|
||||
| *Current situation*: The app is running in a Virtual Machine (VM).
|
||||
| *Suggested outcome*: Deploy it as an OpenShift service.
|
||||
| *Reason(s)*: Automated deployments and we already have an OpenShift cluster.
|
||||
| *Current situation*: The app is running in a Virtual Machine (VM).
|
||||
| *Suggested outcome*: Deploy it as an OpenShift service.
|
||||
| *Reason(s)*: Automated deployments and we already have an OpenShift cluster.
|
||||
|
||||
**Documentation**
|
||||
| *Current situation*: Only a brief README file and `SOP <https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/github2fedmsg/>`_ are available.
|
||||
| *Suggested outcome*: It would be great if the documentation was more extensive. It should include instructions about setting up the development environment, contributing guidelines, code of conduct, etc. This is also covered in the `dev environment enhancement doc <https://fedora-arc.readthedocs.io/en/latest/github2fedmsg/enhancing_the_development_environment.html>`_.
|
||||
| *Reason(s)*: Easier onboarding and setting expectations for contributors.
|
||||
| *Current situation*: Only a brief README file and `SOP
|
||||
<https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/github2fedmsg/>`_ are
|
||||
available.
|
||||
| *Suggested outcome*: It would be great if the documentation was more extensive.
|
||||
It should include instructions about setting up the development environment,
|
||||
contributing guidelines, code of conduct, etc. This is also covered in the `dev
|
||||
environment enhancement doc
|
||||
<https://fedora-arc.readthedocs.io/en/latest/github2fedmsg/enhancing_the_development_environment.html>`_.
|
||||
| *Reason(s)*: Easier onboarding and setting expectations for contributors.
|
||||
|
||||
**Porting to Authlib**
|
||||
| *Current outcome*: The authentication library used is `Velruse <https://velruse.readthedocs.io/en/latest/usage.html>`_.
|
||||
| *Suggested using*: `Authlib <https://authlib.org/>`_.
|
||||
| *Reason(s)*: Velruse hasn't been updated in 9 years and Authlib is in active development.
|
||||
| *Current outcome*: The authentication library used is `Velruse
|
||||
<https://velruse.readthedocs.io/en/latest/usage.html>`_.
|
||||
| *Suggested using*: `Authlib <https://authlib.org/>`_.
|
||||
| *Reason(s)*: Velruse hasn't been updated in 9 years and Authlib is in active
|
||||
development.
|
||||
|
||||
**Development environment**
|
||||
| *Current situation*: This is covered in depth in our other document about enhancing the development environment.
|
||||
| *Current situation*: This is covered in depth in our other document about
|
||||
enhancing the development environment.
|
||||
|
||||
Notes
|
||||
-----
|
||||
* setuptools>=58 `breaks support for 2to3 <https://setuptools.pypa.io/en/latest/history.html#v58-0-0>`_.
|
||||
* 2to3 is `deprecated since Python 3.11 and will be removed in Python 3.13 <https://docs.python.org/3/library/2to3.html>`_.
|
||||
|
||||
- setuptools>=58 `breaks support for 2to3
|
||||
<https://setuptools.pypa.io/en/latest/history.html#v58-0-0>`_.
|
||||
- 2to3 is `deprecated since Python 3.11 and will be removed in Python 3.13
|
||||
<https://docs.python.org/3/library/2to3.html>`_.
|
||||
|
||||
The Good and the Bad
|
||||
--------------------
|
||||
|
||||
**Advantages**
|
||||
1. The project is written in (now EOLed) Python 2.7.x
|
||||
and as Python 3 is currently in active development, the project can
|
||||
benefit from the performance improvements, bug fixes, and security patches
|
||||
included in the language itself.
|
||||
2. The project makes use of a certain set of dependencies, the support of
|
||||
which has not been ported from Python 2 to Python 3. Porting into
|
||||
Python 3 would necessitate the use of newer dependencies which would be
|
||||
more updated, secure and maintained.
|
||||
1. The project is written in (now EOLed) Python 2.7.x and as Python 3 is currently
|
||||
in active development, the project can benefit from the performance improvements,
|
||||
bug fixes, and security patches included in the language itself.
|
||||
2. The project makes use of a certain set of dependencies, the support of which has
|
||||
not been ported from Python 2 to Python 3. Porting into Python 3 would
|
||||
necessitate the use of newer dependencies which would be more updated, secure and
|
||||
maintained.
|
||||
|
||||
**Disadvantages**
|
||||
1. The move from Python 2 to Python 3 would require special attention into
|
||||
finding the alternatives for the dependencies which either support only
|
||||
Python 2 or are unmaintained as of 2022. Inability to find those would
|
||||
render the rewrite unsuccessful.
|
||||
2. We will continue using the `Pyramid web framework <https://trypyramid.com/>`_,
|
||||
however the team lacks the expertise with it. This makes the app difficult
|
||||
to maintain and team members would have to upskill in this area.
|
||||
1. The move from Python 2 to Python 3 would require special attention into finding
|
||||
the alternatives for the dependencies which either support only Python 2 or are
|
||||
unmaintained as of 2022. Inability to find those would render the rewrite
|
||||
unsuccessful.
|
||||
2. We will continue using the `Pyramid web framework <https://trypyramid.com/>`_,
|
||||
however the team lacks the expertise with it. This makes the app difficult to
|
||||
maintain and team members would have to upskill in this area.
|
||||
|
|
|
@ -1,111 +1,135 @@
|
|||
.. _rewrite:
|
||||
|
||||
Complete Rewrite of github2fedmsg Application
|
||||
====
|
||||
=============================================
|
||||
|
||||
This document will investigate the possibility of rewriting github2fedmsg from scratch.
|
||||
|
||||
Elements that could use our attention
|
||||
----
|
||||
-------------------------------------
|
||||
|
||||
**Project configuration**
|
||||
| *Currently uses*: Setuptools (`setup.py <https://github.com/fedora-infra/github2fedmsg/blob/develop/setup.py>`_)
|
||||
| *Suggested using*: Poetry (`pyproject.toml <https://python-poetry.org/docs/pyproject/>`_)
|
||||
| *Reason(s)*: Easier dependency management, project configuration, testing information etc.
|
||||
| *Currently uses*: Setuptools (`setup.py
|
||||
<https://github.com/fedora-infra/github2fedmsg/blob/develop/setup.py>`_)
|
||||
| *Suggested using*: Poetry (`pyproject.toml
|
||||
<https://python-poetry.org/docs/pyproject/>`_)
|
||||
| *Reason(s)*: Easier dependency management, project configuration, testing
|
||||
information etc.
|
||||
|
||||
**Backend language**
|
||||
| *Currently uses*: Python 2.7.x (now `EOLed <https://www.python.org/doc/sunset-python-2/>`_)
|
||||
| *Suggested using*: Python 3.10.x (and above)
|
||||
| *Reason(s)*: Benefit from performance improvements, bug fixes, security patches etc. on the language
|
||||
| *Currently uses*: Python 2.7.x (now `EOLed
|
||||
<https://www.python.org/doc/sunset-python-2/>`_)
|
||||
| *Suggested using*: Python 3.10.x (and above)
|
||||
| *Reason(s)*: Benefit from performance improvements, bug fixes, security patches
|
||||
etc. on the language
|
||||
|
||||
**Base framework**
|
||||
| *Currently uses*: `Pyramid <https://trypyramid.com/>`_
|
||||
| *Suggested using*: `Flask <https://flask.palletsprojects.com/en/2.2.x/>`_ or `FastAPI <https://fastapi.tiangolo.com/>`_
|
||||
| *Reason(s)*: More knowledge about the suggested ones in the team, streamlined development, more features etc.
|
||||
| *Currently uses*: `Pyramid <https://trypyramid.com/>`_
|
||||
| *Suggested using*: `Flask <https://flask.palletsprojects.com/en/2.2.x/>`_ or
|
||||
`FastAPI <https://fastapi.tiangolo.com/>`_
|
||||
| *Reason(s)*: More knowledge about the suggested ones in the team, streamlined
|
||||
development, more features etc.
|
||||
|
||||
**Templating implementation**
|
||||
| *Currently uses*: `Mako <https://www.makotemplates.org/>`_
|
||||
| *Suggested using*: `Jinja <https://jinja.palletsprojects.com/en/3.1.x/>`_
|
||||
| *Reason(s)*: More knowledge about the suggested ones in the team, standardized templating etc.
|
||||
| *Currently uses*: `Mako <https://www.makotemplates.org/>`_
|
||||
| *Suggested using*: `Jinja <https://jinja.palletsprojects.com/en/3.1.x/>`_
|
||||
| *Reason(s)*: More knowledge about the suggested ones in the team, standardized
|
||||
templating etc.
|
||||
|
||||
**Authentication library**
|
||||
| *Currently uses*: `Velruse <https://velruse.readthedocs.io/en/latest/usage.html>`_
|
||||
| *Suggested using*: `Authlib <https://authlib.org/>`_
|
||||
| *Reason(s)*: Authlib is actively maintained while Velruse had its last update `9 years ago <https://pypi.org/project/velruse/1.1.1/>`_.
|
||||
| *Currently uses*: `Velruse
|
||||
<https://velruse.readthedocs.io/en/latest/usage.html>`_
|
||||
| *Suggested using*: `Authlib <https://authlib.org/>`_
|
||||
| *Reason(s)*: Authlib is actively maintained while Velruse had its last update `9
|
||||
years ago <https://pypi.org/project/velruse/1.1.1/>`_.
|
||||
|
||||
**Frontend CSS library**
|
||||
| *Currently uses*: `Bootstrap 3.1.1 (Fedora) <https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/bootstrap-3.1.1-fedora>`_
|
||||
| *Suggested using*: Updated version of Bootstrap for eg. `Boostrap 5 <https://getbootstrap.com/docs/5.0/getting-started/introduction/>`_
|
||||
| *Reason(s)*: Web interface looks/feels dated and is not responsive to variable widths.
|
||||
| *Currently uses*: `Bootstrap 3.1.1 (Fedora)
|
||||
<https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/bootstrap-3.1.1-fedora>`_
|
||||
| *Suggested using*: Updated version of Bootstrap for eg. `Boostrap 5
|
||||
<https://getbootstrap.com/docs/5.0/getting-started/introduction/>`_
|
||||
| *Reason(s)*: Web interface looks/feels dated and is not responsive to variable
|
||||
widths.
|
||||
|
||||
**Notification popups**
|
||||
| *Currently uses*: `Messenger 1.4.1 <https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/messenger-1.4.1>`_
|
||||
| *Suggested using*: Updated version of Messenger or any maintained alternative
|
||||
| *Reason(s)*: Library was last added `8 years back <https://github.com/fedora-infra/github2fedmsg/commit/cb98f6c7b74bb50c7cb8ee58aff81033c23300ee>`_ and could use an update.
|
||||
| *Currently uses*: `Messenger 1.4.1
|
||||
<https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/messenger-1.4.1>`_
|
||||
| *Suggested using*: Updated version of Messenger or any maintained alternative
|
||||
| *Reason(s)*: Library was last added `8 years back
|
||||
<https://github.com/fedora-infra/github2fedmsg/commit/cb98f6c7b74bb50c7cb8ee58aff81033c23300ee>`_
|
||||
and could use an update.
|
||||
|
||||
**Interface interactivity**
|
||||
| *Currently uses*: `JQuery 1.11.0 <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/static/jquery-1.11.0.min.js>`_
|
||||
| *Suggested using*: Updated version of JQuery for eg. `JQuery 3.6.1 <https://code.jquery.com/jquery-3.6.1.min.js>`_
|
||||
| *Reason(s)*: Library was last added `9 years back <https://github.com/fedora-infra/github2fedmsg/commit/8985732f1e22a565dfd3ce9964896e9e4f86657e>`_ and could use an update.
|
||||
| *Currently uses*: `JQuery 1.11.0
|
||||
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/static/jquery-1.11.0.min.js>`_
|
||||
| *Suggested using*: Updated version of JQuery for eg. `JQuery 3.6.1
|
||||
<https://code.jquery.com/jquery-3.6.1.min.js>`_
|
||||
| *Reason(s)*: Library was last added `9 years back
|
||||
<https://github.com/fedora-infra/github2fedmsg/commit/8985732f1e22a565dfd3ce9964896e9e4f86657e>`_
|
||||
and could use an update.
|
||||
|
||||
**API modelling**
|
||||
| *Currently uses*: In-place `creation of data structures <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/views/webhooks.py#L247-L253>`_ in worker functions
|
||||
| *Suggested using*: Data validation and typing libraries like `Pydantic <https://pydantic-docs.helpmanual.io/>`_
|
||||
| *Reason(s)*: Better codebase modularity and readability, auto-validation of API data types etc.
|
||||
| *Currently uses*: In-place `creation of data structures
|
||||
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/views/webhooks.py#L247-L253>`_
|
||||
in worker functions
|
||||
| *Suggested using*: Data validation and typing libraries like `Pydantic
|
||||
<https://pydantic-docs.helpmanual.io/>`_
|
||||
| *Reason(s)*: Better codebase modularity and readability, auto-validation of API
|
||||
data types etc.
|
||||
|
||||
**Command line parsing**
|
||||
| *Currently uses*: None. Just making do with `stdio` and `stdout`.
|
||||
| *Suggested using*: `Click <https://click.palletsprojects.com/en/8.1.x/>`_ or any other alternatives
|
||||
| *Reason(s)*: Easy-to-use command line creation, handling of options and commands with auto-generated help etc.
|
||||
| *Currently uses*: None. Just making do with `stdio` and `stdout`.
|
||||
| *Suggested using*: `Click <https://click.palletsprojects.com/en/8.1.x/>`_ or any
|
||||
other alternatives
|
||||
| *Reason(s)*: Easy-to-use command line creation, handling of options and commands
|
||||
with auto-generated help etc.
|
||||
|
||||
The good and the bad
|
||||
----
|
||||
--------------------
|
||||
|
||||
**Advantages**
|
||||
1. In its current state, the project is written in (now EOLed) Python 2.7.x
|
||||
and as Python 3 is currently in active development, the project can
|
||||
benefit from the performance improvements, bug fixes and security patches
|
||||
included in the language itself.
|
||||
2. The project makes use of a certain set of dependencies, the support of
|
||||
which, have not been ported from Python 2 to Python 3. Porting into
|
||||
Python 3 would necessitate the use of newer dependencies which would be
|
||||
more updated, secure and maintained.
|
||||
3. In its current state, the project makes use of the
|
||||
`Pyramid <https://trypyramid.com/>`_ web framework, the expertise of which
|
||||
the team currently lacks, making it unmaintainable right now. A rewrite
|
||||
using libraries and frameworks that the team is acquainted with would help.
|
||||
4. The web interface makes use of non-default
|
||||
`Mako <https://www.makotemplates.org/>`_-based
|
||||
`templates <https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/templates>`_
|
||||
that work right now but due to their obscure nature, it can become
|
||||
difficult to debug the templates later. A rewrite of templates using
|
||||
a standardized
|
||||
`Jinja <https://jinja.palletsprojects.com/en/3.1.x/>`_ format would go a
|
||||
long way.
|
||||
5. The web interface could use updated versions (or replacements) of
|
||||
unmaintained JavaScript and CSS3 dependencies (for now, it has
|
||||
`Bootstrap 3.1.1 <https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/bootstrap-3.1.1-fedora>`_,
|
||||
`Messenger 1.4.1 <https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/messenger-1.4.1>`_,
|
||||
`JQuery 1.11.0 <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/static/jquery-1.11.0.min.js>`_
|
||||
etc.) in order to introduce responsiveness and provide a better look and
|
||||
feel.
|
||||
6. The API model can be standardized with the use of typing and data
|
||||
validation libraries such as
|
||||
`Pydantic <https://pydantic-docs.helpmanual.io/>`_ instead of
|
||||
`forming data structures <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/views/webhooks.py#L247-L253>`_
|
||||
in the worker function itself, making the code more modular and
|
||||
maintainable in the long run.
|
||||
1. In its current state, the project is written in (now EOLed) Python 2.7.x and as
|
||||
Python 3 is currently in active development, the project can benefit from the
|
||||
performance improvements, bug fixes and security patches included in the language
|
||||
itself.
|
||||
2. The project makes use of a certain set of dependencies, the support of which,
|
||||
have not been ported from Python 2 to Python 3. Porting into Python 3 would
|
||||
necessitate the use of newer dependencies which would be more updated, secure and
|
||||
maintained.
|
||||
3. In its current state, the project makes use of the `Pyramid
|
||||
<https://trypyramid.com/>`_ web framework, the expertise of which the team
|
||||
currently lacks, making it unmaintainable right now. A rewrite using libraries
|
||||
and frameworks that the team is acquainted with would help.
|
||||
4. The web interface makes use of non-default `Mako
|
||||
<https://www.makotemplates.org/>`_-based `templates
|
||||
<https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/templates>`_
|
||||
that work right now but due to their obscure nature, it can become difficult to
|
||||
debug the templates later. A rewrite of templates using a standardized `Jinja
|
||||
<https://jinja.palletsprojects.com/en/3.1.x/>`_ format would go a long way.
|
||||
5. The web interface could use updated versions (or replacements) of unmaintained
|
||||
JavaScript and CSS3 dependencies (for now, it has `Bootstrap 3.1.1
|
||||
<https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/bootstrap-3.1.1-fedora>`_,
|
||||
`Messenger 1.4.1
|
||||
<https://github.com/fedora-infra/github2fedmsg/tree/develop/github2fedmsg/static/messenger-1.4.1>`_,
|
||||
`JQuery 1.11.0
|
||||
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/static/jquery-1.11.0.min.js>`_
|
||||
etc.) in order to introduce responsiveness and provide a better look and feel.
|
||||
6. The API model can be standardized with the use of typing and data validation
|
||||
libraries such as `Pydantic <https://pydantic-docs.helpmanual.io/>`_ instead of
|
||||
`forming data structures
|
||||
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/views/webhooks.py#L247-L253>`_
|
||||
in the worker function itself, making the code more modular and maintainable in
|
||||
the long run.
|
||||
|
||||
**Disadvantages**
|
||||
1. The move from Python 2 to Python 3 would require special attention into
|
||||
the finding alternatives for the dependencies which either support only
|
||||
Python 2 or are unmaintained as of 2022. Inability to find those would
|
||||
render the rewrite unsuccessful.
|
||||
2. In the wake of new ways to interact with an API, some features (eg. web
|
||||
interface, when API is suggested to be used by itself) of the current
|
||||
version have become redundant and a 1:1 rewrite would not help as it
|
||||
would frontport those features too.
|
||||
3. While reworking the database model, it is possible that the existing dump
|
||||
cannot be fully imported due to the changes introduced in the database
|
||||
schema to better work the newer reimplementation, thereby potentially
|
||||
causing data loss.
|
||||
1. The move from Python 2 to Python 3 would require special attention into the
|
||||
finding alternatives for the dependencies which either support only Python 2 or
|
||||
are unmaintained as of 2022. Inability to find those would render the rewrite
|
||||
unsuccessful.
|
||||
2. In the wake of new ways to interact with an API, some features (eg. web
|
||||
interface, when API is suggested to be used by itself) of the current version
|
||||
have become redundant and a 1:1 rewrite would not help as it would frontport
|
||||
those features too.
|
||||
3. While reworking the database model, it is possible that the existing dump cannot
|
||||
be fully imported due to the changes introduced in the database schema to better
|
||||
work the newer reimplementation, thereby potentially causing data loss.
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue