170 lines
11 KiB
ReStructuredText
170 lines
11 KiB
ReStructuredText
.. _enhancing_the_development_environment:
|
|
|
|
Enhancing the Development Environment
|
|
========================================
|
|
|
|
The current state of the development environment of the github2fedmsg project
|
|
leaves a lot to be desired. This document attempts to address all the existing
|
|
tech debts bound to the github2fedmsg project repository.
|
|
|
|
Investigations
|
|
----
|
|
|
|
**Continuous Integration**
|
|
1. Apparently, there has been a continuous integration configured for the
|
|
repository, which was last active on Jun 17, 2020 on
|
|
`PR #27 <https://github.com/fedora-infra/github2fedmsg/pull/27>`_.
|
|
This continuous integration was set up on a Jenkins instance of Fedora
|
|
Infrastructure on the CentOS CI namespace (as deduced by the `details
|
|
URL <https://jenkins-fedora-apps.apps.ci.centos.org/job/github2fedmsg/job/PR-27/1/display/redirect>`_)
|
|
but is either no longer available or can only be accessed via the Red Hat
|
|
VPN. If the former is the case, we would want to reconfigure the
|
|
continuous integration in the project repository but if latter is the
|
|
case, we would want to ensure that the continuous integration is
|
|
accessible to the wider community to promote contributor participation in
|
|
the project.
|
|
2. In order to set up a continuous integration in the project repository,
|
|
`GitHub Actions <https://github.com/features/actions>`_ can be put into
|
|
use as the project repository itself is hosted on GitHub, thus allowing
|
|
for integration and support to a greater extent. It is also possible to
|
|
use Fedora Linux containers in GitHub Actions, as evidenced by their use
|
|
in `Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/.github/workflows/main.yml#L10>`_,
|
|
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/.github/workflows/main.yaml#L8>`_,
|
|
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/.github/workflows/main.yaml#L8>`_,
|
|
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/.github/workflows/main.yml#L9>`_
|
|
and many more such projects, which would help replicate the production
|
|
environment accurately for the relevant build and test jobs. Other
|
|
continuous integration options like `Zuul CI <https://zuul-ci.org/>`_,
|
|
`Travis CI <https://www.travis-ci.com/>`_ etc. can also be considered on
|
|
the basis of particular needs.
|
|
|
|
**Development Environment**
|
|
1. A small amount of information about setting up a virtual environment and
|
|
installing the project using
|
|
`setuptools <https://setuptools.pypa.io/en/latest/>`_
|
|
is currently available on the
|
|
`README.rst document <https://github.com/fedora-infra/github2fedmsg/blob/develop/README.rst>`_
|
|
in the project repository. Although, the current setup works fine in a
|
|
specific environment having (now EOLed) Python 2.7.x, it can be useful to
|
|
abstract the use of setuptools, using dependency management tools like
|
|
`poetry <https://python-poetry.org/>`_,
|
|
`pipenv <https://pipenv.pypa.io/en/latest/>`_ etc.
|
|
to steer away from the unnecessary attention in creating and updating
|
|
the setuptools configuration, facilitating for automatic dependency
|
|
updates and maintaining a list of dependencies specific to development,
|
|
testing and production scenarios.
|
|
2. Maintaining codebase to uphold established quality standards will be of
|
|
great help, both, for the convenience of existing contributors and to
|
|
help onboard potential ones. In the current setup,
|
|
`flake8 <https://flake8.pycqa.org/en/latest/>`_ is
|
|
`used <https://github.com/fedora-infra/github2fedmsg/blob/develop/tox.ini#L13>`_
|
|
for linting the codebase. In addition to flake8,
|
|
`isort <https://isort.readthedocs.io/en/latest/>`_ and
|
|
`black <https://black.readthedocs.io/en/stable/>`_ can
|
|
be used for sorting of dependency imports and enforcing code styles
|
|
wherever necessary. This has been of great help in maintaining code
|
|
quality, as evidenced by their uses in
|
|
`Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/tox.ini#L27-L56>`_,
|
|
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/tox.ini#L25-L41>`_,
|
|
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L16-L33>`_,
|
|
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L26-L54>`_
|
|
and many more such projects.
|
|
|
|
**Automatic Dependency Updates**
|
|
1. Projects such as
|
|
`Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/.github/dependabot.yml>`_,
|
|
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/.github/dependabot.yml>`_ and
|
|
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/.github/dependabot.yml>`_
|
|
use `GitHub Dependabot <https://github.com/dependabot>`_ to help
|
|
automatically update the dependency lockfile as and when new updates and
|
|
vulnerability fixes come out for the unexcluded dependencies, matching
|
|
the version requirement mentioned in the project configuration file.
|
|
While the former case requires having a valid GitHub Dependabot
|
|
configuration in the repository, the latter does not and will work
|
|
regardless of having it or not. Being GitHub's own security worker bot,
|
|
it has a greater extent of integration with the project repositories
|
|
available on GitHub.
|
|
2. Projects such as
|
|
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/renovate.json>`_
|
|
and `Anitya <https://github.com/fedora-infra/anitya/blob/master/renovate.json>`_
|
|
use `Renovate <https://github.com/apps/renovate>`_ to help automatically
|
|
update the dependency lockfile as and when new updates or vulnerability
|
|
fixes come out for the unexcluded dependencies, matching the version
|
|
requirement mentioned in the project configuration file. Renovate is
|
|
highly flexible with its configuration, allows for checking of dependency
|
|
lockfile for verifying checksums, organizing custom schedules for updates
|
|
and also supports auto-merging of the dependency update pull requests.
|
|
|
|
**Outdated dependencies**
|
|
1. The project makes use of a certain set of dependencies which are either
|
|
left in a pre-release state or are no longer actively maintained. While
|
|
these dependencies can be put to use in the project in its current state,
|
|
it is very likely that a lot of security vulnerabilities must have flown
|
|
under the radar unpatched, in the absence of features updates, security
|
|
patches and bug fixes. Apart from that, being older libraries/frameworks -
|
|
the availability of updated documentation is likely to be scarce for
|
|
these, thus making the development using these dependencies more
|
|
challenging for the maintainers.
|
|
2. Moving forward, we would want to look into either possible replacing the
|
|
following dependencies or avoiding their use entirely.
|
|
|
|
1. `Velruse <https://pypi.org/project/velruse/>`_, had its last
|
|
update almost *9 years back* in
|
|
`Aug 30, 2013 <https://pypi.org/project/velruse/1.1.1/>`_.
|
|
2. `WebError <https://pypi.org/project/WebError/>`_, had its last update
|
|
almost *6 years back* in
|
|
`Apr 10, 2016 <https://pypi.org/project/WebError/0.13.1/>`_ and the
|
|
`project documentation states <https://github.com/Pylons/weberror/blob/e395f6d0f0becb299b905197c4577d3b74ad40ae/README.rst#notice>`_
|
|
that the project is no longer actively maintained.
|
|
3. `pyramid-mako <https://pypi.org/project/pyramid-mako/>`_ had its last
|
|
update almost *3 years back* in
|
|
`Aug 19, 2019 <https://pypi.org/project/pyramid-mako/1.1.0/>`_.
|
|
4. `transaction <https://pypi.org/project/transaction/>`_ had its last
|
|
update almost *2 years back* in
|
|
`Dec 11, 2020 <https://pypi.org/project/transaction/3.0.1/>`_.
|
|
|
|
**Contributor's Quality of Life**
|
|
1. Certain additional documentations such as CODE OF CONDUCT, CONTRIBUTING
|
|
GUIDELINES etc. can either be added or linked in, to set the expectations
|
|
right for both existing and potential contributors. The wiki feature
|
|
present in the project repository can be preferred to be put into use
|
|
for keeping documentation like these which are less likely to change
|
|
across multiple releases. These documentation are available in
|
|
repositories such as MDAPI
|
|
`[1] <https://github.com/fedora-infra/mdapi/blob/develop/CODE_OF_CONDUCT.md>`_
|
|
`[2] <https://github.com/fedora-infra/mdapi/blob/develop/CONTRIBUTING.md>`_,
|
|
Meetbot Logs
|
|
`[1] <https://github.com/fedora-infra/mote/blob/main/CODE_OF_CONDUCT.md>`_
|
|
`[2] <https://github.com/fedora-infra/mote/blob/main/CONTRIBUTING.md>`_,
|
|
CentOS Duffy
|
|
`[1] <https://github.com/CentOS/duffy/blob/dev/CONTRIBUTING.md>`_ etc.
|
|
and can be included here too.
|
|
2. Tests and code coverage should be added to ensure that the project code
|
|
remains functional over time across the addition of features, fixing of
|
|
bugs, patching of security vulnerabilities etc. The
|
|
`tests <https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/tests.py>`_
|
|
currently available in the project repository have a limited scope and
|
|
are not automated using continuous integration to test out the code
|
|
`on pull requests <https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#pull_request>`_,
|
|
`on pushes <https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#push>`_
|
|
and
|
|
`before deployment workflows <https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#workflow_dispatch>`_.
|
|
These tests and code coverage have been implemented in
|
|
`Meetbot Logs <https://github.com/fedora-infra/mote/blob/main/tox.ini#L7-L19>`_,
|
|
`CentOS Duffy <https://github.com/CentOS/duffy/blob/dev/tox.ini#L7-L18>`_,
|
|
`Fedora Messaging Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L7-L14>`_,
|
|
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L7-L17>`_
|
|
and many more such repositories and are automated using GitHub Actions.
|
|
|
|
**Automating Deployments to OpenShift**
|
|
1. The project currently makes use of semantic versioning for its releases
|
|
and requires manual intervention for deploying every distinct release to
|
|
both `staging <https://apps.fedoraproject.org/github2fedmsg>`_ and
|
|
`production <https://apps.stg.fedoraproject.org/github2fedmsg>`_
|
|
environments. The effort can be reduced by the use of webhooks which
|
|
would help monitor distinct release on the project repository and then
|
|
automatically run deployment scripts to put the new version up on
|
|
the staging environment. For this, GitHub webhooks can be configured to
|
|
be used on the said OpenShift cluster and the production deployment is
|
|
suggested to require manual intervention to make way for additional due
|
|
diligence before deployment.
|