166 lines
11 KiB
ReStructuredText
166 lines
11 KiB
ReStructuredText
.. _enhancing_the_development_environment:
|
|
|
|
Enhancing the Development Environment
|
|
=====================================
|
|
|
|
The current state of the development environment of the github2fedmsg project leaves a
|
|
lot to be desired. This document attempts to address all the existing tech debts bound
|
|
to the github2fedmsg project repository.
|
|
|
|
Investigations
|
|
--------------
|
|
|
|
**Continuous Integration**
|
|
1. Apparently, there has been a continuous integration configured for the
|
|
repository, which was last active on Jun 17, 2020 on `PR #27
|
|
<https://github.com/fedora-infra/github2fedmsg/pull/27>`_. This continuous
|
|
integration was set up on a Jenkins instance of Fedora Infrastructure on the
|
|
CentOS CI namespace (as deduced by the `details URL
|
|
<https://jenkins-fedora-apps.apps.ci.centos.org/job/github2fedmsg/job/PR-27/1/display/redirect>`_)
|
|
but is either no longer available or can only be accessed via the Red Hat VPN. If
|
|
the former is the case, we would want to reconfigure the continuous integration
|
|
in the project repository but if latter is the case, we would want to ensure that
|
|
the continuous integration is accessible to the wider community to promote
|
|
contributor participation in the project.
|
|
2. In order to set up a continuous integration in the project repository, `GitHub
|
|
Actions <https://github.com/features/actions>`_ can be put into use as the
|
|
project repository itself is hosted on GitHub, thus allowing for integration and
|
|
support to a greater extent. It is also possible to use Fedora Linux containers
|
|
in GitHub Actions, as evidenced by their use in `Meetbot Logs
|
|
<https://github.com/fedora-infra/mote/blob/main/.github/workflows/main.yml#L10>`__,
|
|
`CentOS Duffy
|
|
<https://github.com/CentOS/duffy/blob/dev/.github/workflows/main.yaml#L8>`_,
|
|
`Fedora Messaging Notifier
|
|
<https://github.com/fedora-infra/fmn/blob/fmn-next/.github/workflows/main.yaml#L8>`__,
|
|
`MDAPI
|
|
<https://github.com/fedora-infra/mdapi/blob/develop/.github/workflows/main.yml#L9>`_
|
|
and many more such projects, which would help replicate the production
|
|
environment accurately for the relevant build and test jobs. Other continuous
|
|
integration options like `Zuul CI <https://zuul-ci.org/>`_, `Travis CI
|
|
<https://www.travis-ci.com/>`_ etc. can also be considered on the basis of
|
|
particular needs.
|
|
|
|
**Development Environment**
|
|
1. A small amount of information about setting up a virtual environment and
|
|
installing the project using `setuptools
|
|
<https://setuptools.pypa.io/en/latest/>`_ is currently available on the
|
|
`README.rst document
|
|
<https://github.com/fedora-infra/github2fedmsg/blob/develop/README.rst>`_ in the
|
|
project repository. Although, the current setup works fine in a specific
|
|
environment having (now EOLed) Python 2.7.x, it can be useful to abstract the use
|
|
of setuptools, using dependency management tools like `poetry
|
|
<https://python-poetry.org/>`_, `pipenv <https://pipenv.pypa.io/en/latest/>`_
|
|
etc. to steer away from the unnecessary attention in creating and updating the
|
|
setuptools configuration, facilitating for automatic dependency updates and
|
|
maintaining a list of dependencies specific to development, testing and
|
|
production scenarios.
|
|
2. Maintaining codebase to uphold established quality standards will be of great
|
|
help, both, for the convenience of existing contributors and to help onboard
|
|
potential ones. In the current setup, `flake8
|
|
<https://flake8.pycqa.org/en/latest/>`__ is `used
|
|
<https://github.com/fedora-infra/github2fedmsg/blob/develop/tox.ini#L13>`__ for
|
|
linting the codebase. In addition to flake8, `isort
|
|
<https://isort.readthedocs.io/en/latest/>`__ and `black
|
|
<https://black.readthedocs.io/en/stable/>`__ can be used for sorting of
|
|
dependency imports and enforcing code styles wherever necessary. This has been of
|
|
great help in maintaining code quality, as evidenced by their uses in `Meetbot
|
|
Logs <https://github.com/fedora-infra/mote/blob/main/tox.ini#L27-L56>`__, `CentOS
|
|
Duffy <https://github.com/CentOS/duffy/blob/dev/tox.ini#L25-L41>`__, `Fedora
|
|
Messaging Notifier
|
|
<https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L16-L33>`__, `MDAPI
|
|
<https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L26-L54>`__ and many
|
|
more such projects.
|
|
|
|
**Automatic Dependency Updates**
|
|
1. Projects such as `Meetbot Logs
|
|
<https://github.com/fedora-infra/mote/blob/main/.github/dependabot.yml>`__,
|
|
`CentOS Duffy
|
|
<https://github.com/CentOS/duffy/blob/dev/.github/dependabot.yml>`__ and `MDAPI
|
|
<https://github.com/fedora-infra/mdapi/blob/develop/.github/dependabot.yml>`__
|
|
use `GitHub Dependabot <https://github.com/dependabot>`__ to help automatically
|
|
update the dependency lockfile as and when new updates and vulnerability fixes
|
|
come out for the unexcluded dependencies, matching the version requirement
|
|
mentioned in the project configuration file. While the former case requires
|
|
having a valid GitHub Dependabot configuration in the repository, the latter does
|
|
not and will work regardless of having it or not. Being GitHub's own security
|
|
worker bot, it has a greater extent of integration with the project repositories
|
|
available on GitHub.
|
|
2. Projects such as `Fedora Messaging Notifier
|
|
<https://github.com/fedora-infra/fmn/blob/fmn-next/renovate.json>`__ and `Anitya
|
|
<https://github.com/fedora-infra/anitya/blob/master/renovate.json>`_ use
|
|
`Renovate <https://github.com/apps/renovate>`_ to help automatically update the
|
|
dependency lockfile as and when new updates or vulnerability fixes come out for
|
|
the unexcluded dependencies, matching the version requirement mentioned in the
|
|
project configuration file. Renovate is highly flexible with its configuration,
|
|
allows for checking of dependency lockfile for verifying checksums, organizing
|
|
custom schedules for updates and also supports auto-merging of the dependency
|
|
update pull requests.
|
|
|
|
**Outdated dependencies**
|
|
1. The project makes use of a certain set of dependencies which are either left in a
|
|
pre-release state or are no longer actively maintained. While these dependencies
|
|
can be put to use in the project in its current state, it is very likely that a
|
|
lot of security vulnerabilities must have flown under the radar unpatched, in the
|
|
absence of features updates, security patches and bug fixes. Apart from that,
|
|
being older libraries/frameworks - the availability of updated documentation is
|
|
likely to be scarce for these, thus making the development using these
|
|
dependencies more challenging for the maintainers.
|
|
2. Moving forward, we would want to look into either possible replacing the
|
|
following dependencies or avoiding their use entirely.
|
|
|
|
1. `Velruse <https://pypi.org/project/velruse/>`_, had its last update almost *9
|
|
years back* in `Aug 30, 2013 <https://pypi.org/project/velruse/1.1.1/>`_.
|
|
2. `WebError <https://pypi.org/project/WebError/>`_, had its last update almost
|
|
*6 years back* in `Apr 10, 2016 <https://pypi.org/project/WebError/0.13.1/>`_
|
|
and the `project documentation states
|
|
<https://github.com/Pylons/weberror/blob/e395f6d0f0becb299b905197c4577d3b74ad40ae/README.rst#notice>`_
|
|
that the project is no longer actively maintained.
|
|
3. `pyramid-mako <https://pypi.org/project/pyramid-mako/>`_ had its last update
|
|
almost *3 years back* in `Aug 19, 2019
|
|
<https://pypi.org/project/pyramid-mako/1.1.0/>`_.
|
|
4. `transaction <https://pypi.org/project/transaction/>`_ had its last update
|
|
almost *2 years back* in `Dec 11, 2020
|
|
<https://pypi.org/project/transaction/3.0.1/>`_.
|
|
|
|
**Contributor's Quality of Life**
|
|
1. Certain additional documentations such as CODE OF CONDUCT, CONTRIBUTING
|
|
GUIDELINES etc. can either be added or linked in, to set the expectations right
|
|
for both existing and potential contributors. The wiki feature present in the
|
|
project repository can be preferred to be put into use for keeping documentation
|
|
like these which are less likely to change across multiple releases. These
|
|
documentation are available in repositories such as MDAPI `[1]
|
|
<https://github.com/fedora-infra/mdapi/blob/develop/CODE_OF_CONDUCT.md>`__ `[2]
|
|
<https://github.com/fedora-infra/mdapi/blob/develop/CONTRIBUTING.md>`__, Meetbot
|
|
Logs `[1] <https://github.com/fedora-infra/mote/blob/main/CODE_OF_CONDUCT.md>`__
|
|
`[2] <https://github.com/fedora-infra/mote/blob/main/CONTRIBUTING.md>`__, CentOS
|
|
Duffy `[1] <https://github.com/CentOS/duffy/blob/dev/CONTRIBUTING.md>`__ etc. and
|
|
can be included here too.
|
|
2. Tests and code coverage should be added to ensure that the project code remains
|
|
functional over time across the addition of features, fixing of bugs, patching of
|
|
security vulnerabilities etc. The `tests
|
|
<https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/tests.py>`_
|
|
currently available in the project repository have a limited scope and are not
|
|
automated using continuous integration to test out the code `on pull requests
|
|
<https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#pull_request>`_,
|
|
`on pushes
|
|
<https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#push>`_
|
|
and `before deployment workflows
|
|
<https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#workflow_dispatch>`_.
|
|
These tests and code coverage have been implemented in `Meetbot Logs
|
|
<https://github.com/fedora-infra/mote/blob/main/tox.ini#L7-L19>`__, `CentOS Duffy
|
|
<https://github.com/CentOS/duffy/blob/dev/tox.ini#L7-L18>`__, `Fedora Messaging
|
|
Notifier <https://github.com/fedora-infra/fmn/blob/fmn-next/tox.ini#L7-L14>`__,
|
|
`MDAPI <https://github.com/fedora-infra/mdapi/blob/develop/tox.ini#L7-L17>`__ and
|
|
many more such repositories and are automated using GitHub Actions.
|
|
|
|
**Automating Deployments to OpenShift**
|
|
1. The project currently makes use of semantic versioning for its releases and
|
|
requires manual intervention for deploying every distinct release to both
|
|
`staging <https://apps.fedoraproject.org/github2fedmsg>`_ and `production
|
|
<https://apps.stg.fedoraproject.org/github2fedmsg>`_ environments. The effort can
|
|
be reduced by the use of webhooks which would help monitor distinct release on
|
|
the project repository and then automatically run deployment scripts to put the
|
|
new version up on the staging environment. For this, GitHub webhooks can be
|
|
configured to be used on the said OpenShift cluster and the production deployment
|
|
is suggested to require manual intervention to make way for additional due
|
|
diligence before deployment.
|