Update the developer doc
Mention the cookiecutter template. Signed-off-by: Aurélien Bompard <aurelien@bompard.org>
This commit is contained in:
Aurélien Bompard
parent
7713f540bb
commit
66fcba764c
6 changed files with 136 additions and 229 deletions
|
|
@ -78,9 +78,20 @@ Pyramid extension that uses OAuthlib. It does not appear to be actively
|
|||
maintained, but it is a reasonable starting point for our few Pyramid
|
||||
applications.
|
||||
|
||||
==== Authlib
|
||||
|
||||
https://authlib.org/[Authlib] is a Python library to build OAuth and
|
||||
OpenID Connect servers and clients. It has integration layers for Flask
|
||||
and Django as protocol servers and clients, as well as client integration
|
||||
layers for Requests, HTTPX, Starlette and FastAPI. As a client app
|
||||
authenticating with OpenID Connect, it is most useful if several
|
||||
authentication providers are to be supported (such as social networks).
|
||||
|
||||
==== Flask-OIDC
|
||||
|
||||
link:#flask-oidc[Flask-OIDC] is a Flask extension.
|
||||
https://github.com/fedora-infra/flask-oidc/[Flask-OIDC] is a Flask
|
||||
extension that makes using Authlib easier when using Flask with only one
|
||||
authentication provider (such as Fedora's identity provider).
|
||||
|
||||
==== Mozilla-Django-OIDC
|
||||
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ head as they move from project to project.
|
|||
We follow the https://www.python.org/dev/peps/pep-0008/[PEP8] style
|
||||
guide for Python. Projects should make it easy to check new code for
|
||||
PEP8 violations preferably by
|
||||
https://pypi.python.org/pypi/flake8[flake8]. It is up to the individual
|
||||
https://docs.astral.sh/ruff/[Ruff]. It is up to the individual
|
||||
project to choose an enforcement method, but it should be clearly
|
||||
documented and continuous integration tests should ensure code is
|
||||
correctly styled before merging pull requests.
|
||||
|
|
@ -21,70 +21,18 @@ example, the maximum line length might vary. The test suite should
|
|||
enforce this.
|
||||
====
|
||||
|
||||
==== Enforcement
|
||||
==== Linting
|
||||
|
||||
Projects should automatically enforce code style. How a project does so
|
||||
is up to the maintainers, but several good options are documented here.
|
||||
Ruff is capable of running a lot of rules to lint the source code.
|
||||
The categories of rules to be run can be selecting by
|
||||
https://docs.astral.sh/ruff/configuration/[configuring ruff] in the
|
||||
`pyproject.toml` file.
|
||||
|
||||
===== Tox
|
||||
A reasonable set of rules and configuration can be found in the
|
||||
Cookiecutter template's
|
||||
https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/pyproject.toml#L77[pyproject.toml]
|
||||
file.
|
||||
|
||||
`tox-config` is an excellent way to test style. `flake8` looks in, among
|
||||
other places, `tox.ini` for its configuration so if you're already using
|
||||
tox this is a good place to place your configuration. For example,
|
||||
adding the following snippet to your `tox.ini` file will run `flake8` as
|
||||
part of your test suite:
|
||||
|
||||
....
|
||||
[testenv:lint]
|
||||
deps =
|
||||
flake8 > 3.0
|
||||
commands =
|
||||
python -m flake8 {posargs}
|
||||
|
||||
[flake8]
|
||||
show-source = True
|
||||
max-line-length = 100
|
||||
exclude = .git,.tox,dist,*egg
|
||||
....
|
||||
|
||||
===== Unit Test
|
||||
|
||||
If you're not using `tox`, you can add a unit test to your test suite:
|
||||
|
||||
....
|
||||
"""This module runs flake8 on a subset of the code base"""
|
||||
import os
|
||||
import subprocess
|
||||
import unittest
|
||||
|
||||
# Adjust this as necessary to ensure REPO_PATH resolves to the root of your
|
||||
# repository.
|
||||
REPO_PATH = os.path.abspath(os.path.dirname(os.path.join(os.path.dirname(__file__), '../')))
|
||||
|
||||
class TestStyle(unittest.TestCase):
|
||||
"""Run flake8 on the repository directory as part of the unit tests."""
|
||||
|
||||
def test_code_with_flake8(self):
|
||||
"""Assert the code is PEP8-compliant"""
|
||||
|
||||
# enforced_paths = [
|
||||
# 'mypackage/pep8subpackage/',
|
||||
# 'mypackage/a_module.py',
|
||||
# ]
|
||||
# enforced_paths = [os.path.join(REPO_PATH, p) for p in enforced_paths]
|
||||
|
||||
# If your entire codebase is not already PEP8 compliant, you can enforce
|
||||
# the style incrementally using ``enforced_paths``.
|
||||
#
|
||||
# flake8_command = ['flake8', '--max-line-length', '100'] + enforced_paths
|
||||
flake8_command = ['flake8', '--max-line-length', '100', REPO_PATH]
|
||||
|
||||
self.assertEqual(subprocess.call(flake8_command), 0)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
....
|
||||
|
||||
==== Auto formatting
|
||||
|
||||
|
|
@ -107,18 +55,56 @@ Your text editor is very likely to have a plugin to run Black on file
|
|||
saving. The documentation has instructions to set it up in Vim and in VS
|
||||
Code (and in Emacs).
|
||||
|
||||
You can check that your code is properly formatted according to Black's
|
||||
settings by adding the following snippet to your `tox.ini` file:
|
||||
|
||||
==== Enforcement
|
||||
|
||||
Projects should automatically enforce code style. How a project does so
|
||||
is up to the maintainers, but several good options are documented here.
|
||||
|
||||
===== Tox
|
||||
|
||||
https://tox.wiki[Tox] is an excellent way to test style. For example,
|
||||
adding the following snippet to your `tox.ini` file will run `ruff` as
|
||||
part of your test suite:
|
||||
|
||||
....
|
||||
[testenv:lint]
|
||||
deps =
|
||||
ruff
|
||||
commands =
|
||||
python -m ruff check {posargs:.}
|
||||
|
||||
[testenv:format]
|
||||
deps =
|
||||
black
|
||||
commands =
|
||||
python -m black --check {posargs:.}
|
||||
|
||||
....
|
||||
|
||||
Remember to add `format` to your Tox `envlist`.
|
||||
Remember to add `lint` and `format` to your Tox `envlist`.
|
||||
|
||||
|
||||
===== Pre-Commit
|
||||
|
||||
https://pre-commit.com/[Pre-commit] is a git hook that will run before Git
|
||||
stores a commit. It has the ability to run a set of tools against the files
|
||||
that are to be committed, and thus will let the developer know of errors and
|
||||
violations early in the development process. Install the `pre-commit` package
|
||||
and run `pre-commit install` to setup the hook.
|
||||
|
||||
The tools to be run are listed in a `.pre-commit-config.yaml` file at the top
|
||||
of the repository. You can find an example in our CookieCutter template's
|
||||
https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/.pre-commit-config.yaml[.pre-commit-config.yaml]
|
||||
file.
|
||||
|
||||
|
||||
It is also possible and recommended to run the pre-commit checks as part of
|
||||
the test suite in tox, see for example the `checks` section in our
|
||||
CookieCutter template's
|
||||
https://github.com/fedora-infra/cookiecutter-python-app/blob/5ad9adb75600ac13fc65bb9fd847e5091a5d983b/%7B%7B%20cookiecutter.slug%20%7D%7D/tox.ini#L34[tox.ini]
|
||||
file.
|
||||
|
||||
|
||||
=== Javascript
|
||||
|
||||
|
|
|
|||
|
|
@ -108,51 +108,35 @@ not conflict by writing in the same changelog file, and a link to the
|
|||
issue, the pull request or the commit is automatically inserted.
|
||||
|
||||
In your project root, add a `pyproject.toml` file with a
|
||||
`tool.towncrier` section similar to the one in
|
||||
https://raw.githubusercontent.com/fedora-infra/fedora-messaging/master/pyproject.toml[fedora-messaging].
|
||||
`tool.towncrier` section similar to the one in our CookieCutter template's
|
||||
https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/pyproject.toml[pyproject.toml]
|
||||
file.
|
||||
|
||||
Create a `news` directory where the news items will be written, and in
|
||||
there create a `_template.rst` file with a content similar to the one in
|
||||
https://raw.githubusercontent.com/fedora-infra/fedora-messaging/master/news/_template.rst[fedora-messaging].
|
||||
|
||||
Of course, replace `fedora_messaging` and the project URL by yours,
|
||||
where applicable.
|
||||
Create a `changelog.d` directory where the changelog items will be written,
|
||||
and inthere create a `_template.md` file with a content similar to the one in
|
||||
https://raw.githubusercontent.com/fedora-infra/cookiecutter-python-app/main/%7B%7B%20cookiecutter.slug%20%7D%7D/changelog.d/_template.md[our template].
|
||||
|
||||
Of course, replace the first section (the `reference` macro) with the
|
||||
project's Github URL. Or just use the CookieCutter template.
|
||||
|
||||
Then create a `docs/changelog.rst` file (location configured in the
|
||||
`pyproject.toml` file) with the follwing content:
|
||||
`pyproject.toml` file) with a content similar to:
|
||||
|
||||
....
|
||||
=============
|
||||
Release Notes
|
||||
=============
|
||||
# Release Notes
|
||||
|
||||
.. towncrier release notes start
|
||||
<!-- towncrier release notes start -->
|
||||
....
|
||||
|
||||
Then each commit can add a file in the `news` folder to document the
|
||||
change. The file has the `source.type` name format, where `source` is one
|
||||
of:
|
||||
|
||||
* `42` when the change is described in issue `42`
|
||||
* `PR42` when the change has been implemented in pull request `42`, and
|
||||
there is no associated issue
|
||||
* `Cabcdef` when the change has been implemented in changeset `abcdef`,
|
||||
and there is no associated issue or pull request.
|
||||
* `username` for contributors (`author` extention). It should be the
|
||||
username part of their commits' email address.
|
||||
|
||||
And where the `type` part of the filename is:
|
||||
|
||||
* `feature`: for new features
|
||||
* `bug`: for bug fixes
|
||||
* `api`: for API changes
|
||||
* `dev`: for development-related changes
|
||||
* `author`: for contributor names
|
||||
* `other`: for other changes
|
||||
Then each commit can add a file in the `changelog.d` folder to document the
|
||||
change. The format is described in our CookieCutter template's
|
||||
https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/docs/contributing.rst#release-notes[contributing]
|
||||
document.
|
||||
|
||||
A preview of the release notes can be generated with
|
||||
`towncrier --draft`.
|
||||
`towncrier build --draft`.
|
||||
|
||||
When running `towncrier`, the tool will write the changelog file and
|
||||
remove the individual news fragments. These changes can then be
|
||||
When running `towncrier build`, the tool will write the changelog file and
|
||||
remove the individual changelog item files. These changes can then be
|
||||
committed as part of the release commit.
|
||||
|
|
|
|||
|
|
@ -33,6 +33,12 @@ with SQLAlchemy. It will configure a scoped session for you, set up a
|
|||
declarative base class, and provide a convenient
|
||||
`flask_sqlalchemy.BaseQuery` sub-class for you.
|
||||
|
||||
===== SQLAlchemy Helpers
|
||||
|
||||
https://github.com/fedora-infra/sqlalchemy-helpers/[SQLAlchemy-Helpers] is an
|
||||
alternative to using Flask-SQLAlchemy if you need your models to be usable
|
||||
from outside Flask.
|
||||
|
||||
==== SQLAlchemy
|
||||
|
||||
http://www.sqlalchemy.org/[SQLAlchemy] is an SQL toolkit and Object
|
||||
|
|
|
|||
|
|
@ -26,6 +26,10 @@ configuration. Any exclusion lines that appear in the codebase MUST be
|
|||
sufficiently explained. Code that is only executed during test suite
|
||||
runs MAY be exempted from this check.
|
||||
|
||||
Note that https://docs.astral.sh/ruff/[Ruff] is also capable of running
|
||||
security checks equivalent to Bandit's. Make sure you are selecting the
|
||||
`S` category in `pyproject.toml`.
|
||||
|
||||
=== Authentication
|
||||
|
||||
The application MUST use OpenID Connect to authenticate users. The
|
||||
|
|
|
|||
|
|
@ -85,61 +85,13 @@ interpreters. It also allows you to define arbitrary test environments,
|
|||
so it's an excellent place to run the code style tests and to ensure the
|
||||
project's documentation builds without errors or warnings.
|
||||
|
||||
Here's an example `tox.ini` file that runs a project's unit tests in
|
||||
Python 2.7, Python 3.4, Python 3.5, and Python 3.6. It also runs
|
||||
https://pypi.python.org/pypi/flake8[flake8] on the entire codebase and
|
||||
builds the documentation with the `warnings treated as errors` Sphinx
|
||||
flag enabled. Finally, it enforces 100% coverage on lines edited by new
|
||||
patches using https://pypi.org/project/diff-cover/[diff-cover]:
|
||||
You can find an example in our CookieCutter template's
|
||||
https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/tox.ini[tox.ini].
|
||||
It runs the test suite in various versions of Python, reports the coverage,
|
||||
runs the pre-commit checks, makes sure we're not depending on software with
|
||||
forbidden licenses, and builds the documentation with the `warnings treated
|
||||
as errors` Sphinx flag enabled.
|
||||
|
||||
....
|
||||
[tox]
|
||||
envlist = py27,py34,py35,py36,lint,diff-cover,docs
|
||||
# If the user is missing an interpreter, don't fail
|
||||
skip_missing_interpreters = True
|
||||
|
||||
[testenv]
|
||||
deps =
|
||||
-rtest-requirements.txt
|
||||
# Substitute your test runner of choice
|
||||
commands =
|
||||
py.test
|
||||
# When running in OpenShift you don't have a username, so expanduser
|
||||
# won't work. If you are running your tests in CentOS CI, this line is
|
||||
# important so the tests can pass there, otherwise tox will fail to find
|
||||
# a home directory when looking for configuration files.
|
||||
passenv = HOME
|
||||
|
||||
[testenv:diff-cover]
|
||||
deps =
|
||||
diff-cover
|
||||
commands =
|
||||
diff-cover coverage.xml --compare-branch=origin/master --fail-under=100
|
||||
|
||||
[testenv:docs]
|
||||
changedir = docs
|
||||
deps =
|
||||
sphinx
|
||||
sphinxcontrib-httpdomain
|
||||
-rrequirements.txt
|
||||
whitelist_externals =
|
||||
mkdir
|
||||
sphinx-build
|
||||
commands=
|
||||
mkdir -p _static
|
||||
sphinx-build -W -b html -d {envtmpdir}/doctrees . _build/html
|
||||
|
||||
[testenv:lint]
|
||||
deps =
|
||||
flake8 > 3.0
|
||||
commands =
|
||||
python -m flake8 {posargs}
|
||||
|
||||
[flake8]
|
||||
show-source = True
|
||||
max-line-length = 100
|
||||
exclude = .git,.tox,dist,*egg
|
||||
....
|
||||
|
||||
=== Coverage
|
||||
|
||||
|
|
@ -147,53 +99,39 @@ https://pypi.python.org/pypi/coverage/[coverage] is a good way to
|
|||
collect test coverage statistics.
|
||||
http://docs.pytest.org/en/latest/contents.html[pytest] has a
|
||||
https://pypi.python.org/pypi/pytest-cov[pytest-cov] plugin that
|
||||
integrates with https://pypi.python.org/pypi/coverage/[coverage] and
|
||||
https://pypi.python.org/pypi/nose-cov[nose-cov] provides integration for
|
||||
the https://nose.readthedocs.io/en/latest/[nose] test runner.
|
||||
https://pypi.org/project/diff-cover/[diff-cover] can be used to ensure
|
||||
integrates with https://pypi.python.org/pypi/coverage/[coverage].
|
||||
https://pypi.org/project/diff-cover/[Diff-Cover] can be used to ensure
|
||||
that all lines edited in a patch have coverage.
|
||||
|
||||
It's possible (and recommended) to have the test suite fail if the
|
||||
coverage percentage goes down. This example `.coveragerc`:
|
||||
coverage is below 100%. See the template's
|
||||
https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/pyproject.toml#L98[pyproject.toml]
|
||||
file for an example.
|
||||
|
||||
An alternative to having 100% coverage is to forbid new changes from bringing
|
||||
the coverage down. This can be checked by `diff-cover` with the following
|
||||
`tox.ini` snippet:
|
||||
|
||||
....
|
||||
[run]
|
||||
# Track what conditional branches are covered.
|
||||
branch = True
|
||||
include =
|
||||
my_python_package/*
|
||||
[tox]
|
||||
envlist = pyXX...,diff-cover,...
|
||||
# If the user is missing an interpreter, don't fail
|
||||
skip_missing_interpreters = True
|
||||
|
||||
[report]
|
||||
# Fail if the coverage is not 100%
|
||||
fail_under = 100
|
||||
# Display results with up 1/100th of a percent accuracy.
|
||||
precision = 2
|
||||
exclude_lines =
|
||||
pragma: no cover
|
||||
|
||||
# Don't complain if tests don't hit defensive assertion code
|
||||
raise AssertionError
|
||||
raise NotImplementedError
|
||||
|
||||
if __name__ == .__main__.:
|
||||
omit =
|
||||
my_python_package/tests/*
|
||||
[testenv:diff-cover]
|
||||
deps =
|
||||
diff-cover
|
||||
commands =
|
||||
diff-cover coverage.xml --compare-branch=origin/develop --fail-under=100
|
||||
....
|
||||
|
||||
To configure `pytest` to collect coverage data on your project, edit
|
||||
`setup.cfg` and add this block, substituting `yourpackage` with the name
|
||||
of the Python package you are measuring coverage on:
|
||||
You must then make sure that the `pytest` command line contains
|
||||
`--cov-report xml` to produce the `coverage.xml` file that `diff-cover` will
|
||||
analyze.
|
||||
|
||||
....
|
||||
[tool:pytest]
|
||||
addopts = --cov-config .coveragerc --cov=yourpackage --cov-report term --cov-report xml --cov-report html
|
||||
....
|
||||
|
||||
This causes coverage (and any test running plugins using coverage) to fail if
|
||||
the coverage level is not 100%. New projects should enforce 100% test
|
||||
coverage. Existing projects should ensure test coverage does not drop to
|
||||
accept a pull request and should increase the minimum test coverage
|
||||
until it is 100%.
|
||||
New projects should enforce 100% test coverage. Existing projects should
|
||||
ensure test coverage does not drop to accept a pull request and should
|
||||
increase the minimum test coverage until it is 100%.
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
|
|
@ -214,46 +152,20 @@ dependencies are checked recursively.
|
|||
|
||||
The licenses are validated against a set of acceptable licenses that you
|
||||
define in a file called `.license_strategy.ini` in your project
|
||||
directory. Here is an example of such a file, that would accept Free
|
||||
licenses:
|
||||
directory. To avoid having to maintain and sync the same file in every
|
||||
project, a shared file is available
|
||||
https://raw.githubusercontent.com/fedora-infra/shared/main/liccheck-strategy.ini[in the shared repo].
|
||||
|
||||
....
|
||||
[Licenses]
|
||||
authorized_licenses:
|
||||
bsd
|
||||
new bsd
|
||||
simplified bsd
|
||||
apache
|
||||
apache 2.0
|
||||
apache software
|
||||
gnu lgpl
|
||||
gpl v2
|
||||
gpl v3
|
||||
lgpl with exceptions or zpl
|
||||
isc
|
||||
isc license (iscl)
|
||||
mit
|
||||
python software foundation
|
||||
zpl 2.1
|
||||
....
|
||||
The CookieCutter template contains
|
||||
https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/devel/run-liccheck.sh[a shell script]
|
||||
that will download the shared file and check the dependencies' licenses
|
||||
against it.
|
||||
|
||||
The verification is case-insensitive, and is done on both the `license`
|
||||
and the `classifiers` metadata fields. See
|
||||
https://pypi.org/project/liccheck/[liccheck]'s documentation for more
|
||||
details.
|
||||
If you want to add a license to the list, feel free to open a Pull Request
|
||||
against https://github.com/fedora-infra/shared[the shared repo].
|
||||
|
||||
You can automate the license check with the following snippet in your
|
||||
`tox.ini` file:
|
||||
You can automate the license check by running the shell script in Tox.
|
||||
|
||||
....
|
||||
[testenv:licenses]
|
||||
deps =
|
||||
liccheck
|
||||
commands =
|
||||
liccheck -s .license_strategy.ini
|
||||
....
|
||||
|
||||
Remember to add `licenses` to your Tox `envlist`.
|
||||
|
||||
=== Security
|
||||
|
||||
|
|
@ -271,3 +183,7 @@ commands =
|
|||
....
|
||||
|
||||
Remember to add `bandit` to your Tox `envlist`.
|
||||
|
||||
Note that https://docs.astral.sh/ruff/[Ruff] is also capable of running
|
||||
security checks equivalent to Bandit's. Make sure you are selecting the
|
||||
`S` category in `pyproject.toml`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue