diff --git a/modules/developer_guide/pages/auth.adoc b/modules/developer_guide/pages/auth.adoc index 2527be9..ec72a56 100644 --- a/modules/developer_guide/pages/auth.adoc +++ b/modules/developer_guide/pages/auth.adoc @@ -78,9 +78,20 @@ Pyramid extension that uses OAuthlib. It does not appear to be actively maintained, but it is a reasonable starting point for our few Pyramid applications. +==== Authlib + +https://authlib.org/[Authlib] is a Python library to build OAuth and +OpenID Connect servers and clients. It has integration layers for Flask +and Django as protocol servers and clients, as well as client integration +layers for Requests, HTTPX, Starlette and FastAPI. As a client app +authenticating with OpenID Connect, it is most useful if several +authentication providers are to be supported (such as social networks). + ==== Flask-OIDC -link:#flask-oidc[Flask-OIDC] is a Flask extension. +https://github.com/fedora-infra/flask-oidc/[Flask-OIDC] is a Flask +extension that makes using Authlib easier when using Flask with only one +authentication provider (such as Fedora's identity provider). ==== Mozilla-Django-OIDC diff --git a/modules/developer_guide/pages/code-style.adoc b/modules/developer_guide/pages/code-style.adoc index 59d62ed..7b3b6e5 100644 --- a/modules/developer_guide/pages/code-style.adoc +++ b/modules/developer_guide/pages/code-style.adoc @@ -9,7 +9,7 @@ head as they move from project to project. We follow the https://www.python.org/dev/peps/pep-0008/[PEP8] style guide for Python. Projects should make it easy to check new code for PEP8 violations preferably by -https://pypi.python.org/pypi/flake8[flake8]. It is up to the individual +https://docs.astral.sh/ruff/[Ruff]. It is up to the individual project to choose an enforcement method, but it should be clearly documented and continuous integration tests should ensure code is correctly styled before merging pull requests. @@ -21,70 +21,18 @@ example, the maximum line length might vary. The test suite should enforce this. ==== -==== Enforcement +==== Linting -Projects should automatically enforce code style. How a project does so -is up to the maintainers, but several good options are documented here. +Ruff is capable of running a lot of rules to lint the source code. +The categories of rules to be run can be selecting by +https://docs.astral.sh/ruff/configuration/[configuring ruff] in the +`pyproject.toml` file. -===== Tox +A reasonable set of rules and configuration can be found in the +Cookiecutter template's +https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/pyproject.toml#L77[pyproject.toml] +file. -`tox-config` is an excellent way to test style. `flake8` looks in, among -other places, `tox.ini` for its configuration so if you're already using -tox this is a good place to place your configuration. For example, -adding the following snippet to your `tox.ini` file will run `flake8` as -part of your test suite: - -.... -[testenv:lint] -deps = - flake8 > 3.0 -commands = - python -m flake8 {posargs} - -[flake8] -show-source = True -max-line-length = 100 -exclude = .git,.tox,dist,*egg -.... - -===== Unit Test - -If you're not using `tox`, you can add a unit test to your test suite: - -.... -"""This module runs flake8 on a subset of the code base""" -import os -import subprocess -import unittest - -# Adjust this as necessary to ensure REPO_PATH resolves to the root of your -# repository. -REPO_PATH = os.path.abspath(os.path.dirname(os.path.join(os.path.dirname(__file__), '../'))) - -class TestStyle(unittest.TestCase): - """Run flake8 on the repository directory as part of the unit tests.""" - - def test_code_with_flake8(self): - """Assert the code is PEP8-compliant""" - - # enforced_paths = [ - # 'mypackage/pep8subpackage/', - # 'mypackage/a_module.py', - # ] - # enforced_paths = [os.path.join(REPO_PATH, p) for p in enforced_paths] - - # If your entire codebase is not already PEP8 compliant, you can enforce - # the style incrementally using ``enforced_paths``. - # - # flake8_command = ['flake8', '--max-line-length', '100'] + enforced_paths - flake8_command = ['flake8', '--max-line-length', '100', REPO_PATH] - - self.assertEqual(subprocess.call(flake8_command), 0) - - -if __name__ == '__main__': - unittest.main() -.... ==== Auto formatting @@ -107,18 +55,56 @@ Your text editor is very likely to have a plugin to run Black on file saving. The documentation has instructions to set it up in Vim and in VS Code (and in Emacs). -You can check that your code is properly formatted according to Black's -settings by adding the following snippet to your `tox.ini` file: + +==== Enforcement + +Projects should automatically enforce code style. How a project does so +is up to the maintainers, but several good options are documented here. + +===== Tox + +https://tox.wiki[Tox] is an excellent way to test style. For example, +adding the following snippet to your `tox.ini` file will run `ruff` as +part of your test suite: .... +[testenv:lint] +deps = + ruff +commands = + python -m ruff check {posargs:.} + [testenv:format] deps = black commands = python -m black --check {posargs:.} + .... -Remember to add `format` to your Tox `envlist`. +Remember to add `lint` and `format` to your Tox `envlist`. + + +===== Pre-Commit + +https://pre-commit.com/[Pre-commit] is a git hook that will run before Git +stores a commit. It has the ability to run a set of tools against the files +that are to be committed, and thus will let the developer know of errors and +violations early in the development process. Install the `pre-commit` package +and run `pre-commit install` to setup the hook. + +The tools to be run are listed in a `.pre-commit-config.yaml` file at the top +of the repository. You can find an example in our CookieCutter template's +https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/.pre-commit-config.yaml[.pre-commit-config.yaml] +file. + + +It is also possible and recommended to run the pre-commit checks as part of +the test suite in tox, see for example the `checks` section in our +CookieCutter template's +https://github.com/fedora-infra/cookiecutter-python-app/blob/5ad9adb75600ac13fc65bb9fd847e5091a5d983b/%7B%7B%20cookiecutter.slug%20%7D%7D/tox.ini#L34[tox.ini] +file. + === Javascript diff --git a/modules/developer_guide/pages/documentation.adoc b/modules/developer_guide/pages/documentation.adoc index ad7862b..dc9f524 100644 --- a/modules/developer_guide/pages/documentation.adoc +++ b/modules/developer_guide/pages/documentation.adoc @@ -108,51 +108,35 @@ not conflict by writing in the same changelog file, and a link to the issue, the pull request or the commit is automatically inserted. In your project root, add a `pyproject.toml` file with a -`tool.towncrier` section similar to the one in -https://raw.githubusercontent.com/fedora-infra/fedora-messaging/master/pyproject.toml[fedora-messaging]. +`tool.towncrier` section similar to the one in our CookieCutter template's +https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/pyproject.toml[pyproject.toml] +file. -Create a `news` directory where the news items will be written, and in -there create a `_template.rst` file with a content similar to the one in -https://raw.githubusercontent.com/fedora-infra/fedora-messaging/master/news/_template.rst[fedora-messaging]. -Of course, replace `fedora_messaging` and the project URL by yours, -where applicable. +Create a `changelog.d` directory where the changelog items will be written, +and inthere create a `_template.md` file with a content similar to the one in +https://raw.githubusercontent.com/fedora-infra/cookiecutter-python-app/main/%7B%7B%20cookiecutter.slug%20%7D%7D/changelog.d/_template.md[our template]. + +Of course, replace the first section (the `reference` macro) with the +project's Github URL. Or just use the CookieCutter template. Then create a `docs/changelog.rst` file (location configured in the -`pyproject.toml` file) with the follwing content: +`pyproject.toml` file) with a content similar to: .... -============= -Release Notes -============= +# Release Notes -.. towncrier release notes start + .... -Then each commit can add a file in the `news` folder to document the -change. The file has the `source.type` name format, where `source` is one -of: - -* `42` when the change is described in issue `42` -* `PR42` when the change has been implemented in pull request `42`, and -there is no associated issue -* `Cabcdef` when the change has been implemented in changeset `abcdef`, -and there is no associated issue or pull request. -* `username` for contributors (`author` extention). It should be the -username part of their commits' email address. - -And where the `type` part of the filename is: - -* `feature`: for new features -* `bug`: for bug fixes -* `api`: for API changes -* `dev`: for development-related changes -* `author`: for contributor names -* `other`: for other changes +Then each commit can add a file in the `changelog.d` folder to document the +change. The format is described in our CookieCutter template's +https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/docs/contributing.rst#release-notes[contributing] +document. A preview of the release notes can be generated with -`towncrier --draft`. +`towncrier build --draft`. -When running `towncrier`, the tool will write the changelog file and -remove the individual news fragments. These changes can then be +When running `towncrier build`, the tool will write the changelog file and +remove the individual changelog item files. These changes can then be committed as part of the release commit. diff --git a/modules/developer_guide/pages/frameworks.adoc b/modules/developer_guide/pages/frameworks.adoc index 3b189bc..633e7ab 100644 --- a/modules/developer_guide/pages/frameworks.adoc +++ b/modules/developer_guide/pages/frameworks.adoc @@ -33,6 +33,12 @@ with SQLAlchemy. It will configure a scoped session for you, set up a declarative base class, and provide a convenient `flask_sqlalchemy.BaseQuery` sub-class for you. +===== SQLAlchemy Helpers + +https://github.com/fedora-infra/sqlalchemy-helpers/[SQLAlchemy-Helpers] is an +alternative to using Flask-SQLAlchemy if you need your models to be usable +from outside Flask. + ==== SQLAlchemy http://www.sqlalchemy.org/[SQLAlchemy] is an SQL toolkit and Object diff --git a/modules/developer_guide/pages/security_policy.adoc b/modules/developer_guide/pages/security_policy.adoc index 8aa2ab6..a275a11 100644 --- a/modules/developer_guide/pages/security_policy.adoc +++ b/modules/developer_guide/pages/security_policy.adoc @@ -26,6 +26,10 @@ configuration. Any exclusion lines that appear in the codebase MUST be sufficiently explained. Code that is only executed during test suite runs MAY be exempted from this check. +Note that https://docs.astral.sh/ruff/[Ruff] is also capable of running +security checks equivalent to Bandit's. Make sure you are selecting the +`S` category in `pyproject.toml`. + === Authentication The application MUST use OpenID Connect to authenticate users. The diff --git a/modules/developer_guide/pages/writing-tests.adoc b/modules/developer_guide/pages/writing-tests.adoc index 419cae0..93ac9fa 100644 --- a/modules/developer_guide/pages/writing-tests.adoc +++ b/modules/developer_guide/pages/writing-tests.adoc @@ -85,61 +85,13 @@ interpreters. It also allows you to define arbitrary test environments, so it's an excellent place to run the code style tests and to ensure the project's documentation builds without errors or warnings. -Here's an example `tox.ini` file that runs a project's unit tests in -Python 2.7, Python 3.4, Python 3.5, and Python 3.6. It also runs -https://pypi.python.org/pypi/flake8[flake8] on the entire codebase and -builds the documentation with the `warnings treated as errors` Sphinx -flag enabled. Finally, it enforces 100% coverage on lines edited by new -patches using https://pypi.org/project/diff-cover/[diff-cover]: +You can find an example in our CookieCutter template's +https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/tox.ini[tox.ini]. +It runs the test suite in various versions of Python, reports the coverage, +runs the pre-commit checks, makes sure we're not depending on software with +forbidden licenses, and builds the documentation with the `warnings treated +as errors` Sphinx flag enabled. -.... -[tox] -envlist = py27,py34,py35,py36,lint,diff-cover,docs -# If the user is missing an interpreter, don't fail -skip_missing_interpreters = True - -[testenv] -deps = - -rtest-requirements.txt -# Substitute your test runner of choice -commands = - py.test -# When running in OpenShift you don't have a username, so expanduser -# won't work. If you are running your tests in CentOS CI, this line is -# important so the tests can pass there, otherwise tox will fail to find -# a home directory when looking for configuration files. -passenv = HOME - -[testenv:diff-cover] -deps = - diff-cover -commands = - diff-cover coverage.xml --compare-branch=origin/master --fail-under=100 - -[testenv:docs] -changedir = docs -deps = - sphinx - sphinxcontrib-httpdomain - -rrequirements.txt -whitelist_externals = - mkdir - sphinx-build -commands= - mkdir -p _static - sphinx-build -W -b html -d {envtmpdir}/doctrees . _build/html - -[testenv:lint] -deps = - flake8 > 3.0 -commands = - python -m flake8 {posargs} - -[flake8] -show-source = True -max-line-length = 100 -exclude = .git,.tox,dist,*egg -.... === Coverage @@ -147,53 +99,39 @@ https://pypi.python.org/pypi/coverage/[coverage] is a good way to collect test coverage statistics. http://docs.pytest.org/en/latest/contents.html[pytest] has a https://pypi.python.org/pypi/pytest-cov[pytest-cov] plugin that -integrates with https://pypi.python.org/pypi/coverage/[coverage] and -https://pypi.python.org/pypi/nose-cov[nose-cov] provides integration for -the https://nose.readthedocs.io/en/latest/[nose] test runner. -https://pypi.org/project/diff-cover/[diff-cover] can be used to ensure +integrates with https://pypi.python.org/pypi/coverage/[coverage]. +https://pypi.org/project/diff-cover/[Diff-Cover] can be used to ensure that all lines edited in a patch have coverage. It's possible (and recommended) to have the test suite fail if the -coverage percentage goes down. This example `.coveragerc`: +coverage is below 100%. See the template's +https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/pyproject.toml#L98[pyproject.toml] +file for an example. + +An alternative to having 100% coverage is to forbid new changes from bringing +the coverage down. This can be checked by `diff-cover` with the following +`tox.ini` snippet: .... -[run] -# Track what conditional branches are covered. -branch = True -include = - my_python_package/* +[tox] +envlist = pyXX...,diff-cover,... +# If the user is missing an interpreter, don't fail +skip_missing_interpreters = True -[report] -# Fail if the coverage is not 100% -fail_under = 100 -# Display results with up 1/100th of a percent accuracy. -precision = 2 -exclude_lines = - pragma: no cover - - # Don't complain if tests don't hit defensive assertion code - raise AssertionError - raise NotImplementedError - - if __name__ == .__main__.: -omit = - my_python_package/tests/* +[testenv:diff-cover] +deps = + diff-cover +commands = + diff-cover coverage.xml --compare-branch=origin/develop --fail-under=100 .... -To configure `pytest` to collect coverage data on your project, edit -`setup.cfg` and add this block, substituting `yourpackage` with the name -of the Python package you are measuring coverage on: +You must then make sure that the `pytest` command line contains +`--cov-report xml` to produce the `coverage.xml` file that `diff-cover` will +analyze. -.... -[tool:pytest] -addopts = --cov-config .coveragerc --cov=yourpackage --cov-report term --cov-report xml --cov-report html -.... - -This causes coverage (and any test running plugins using coverage) to fail if -the coverage level is not 100%. New projects should enforce 100% test -coverage. Existing projects should ensure test coverage does not drop to -accept a pull request and should increase the minimum test coverage -until it is 100%. +New projects should enforce 100% test coverage. Existing projects should +ensure test coverage does not drop to accept a pull request and should +increase the minimum test coverage until it is 100%. [NOTE] ==== @@ -214,46 +152,20 @@ dependencies are checked recursively. The licenses are validated against a set of acceptable licenses that you define in a file called `.license_strategy.ini` in your project -directory. Here is an example of such a file, that would accept Free -licenses: +directory. To avoid having to maintain and sync the same file in every +project, a shared file is available +https://raw.githubusercontent.com/fedora-infra/shared/main/liccheck-strategy.ini[in the shared repo]. -.... -[Licenses] -authorized_licenses: - bsd - new bsd - simplified bsd - apache - apache 2.0 - apache software - gnu lgpl - gpl v2 - gpl v3 - lgpl with exceptions or zpl - isc - isc license (iscl) - mit - python software foundation - zpl 2.1 -.... +The CookieCutter template contains +https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/devel/run-liccheck.sh[a shell script] +that will download the shared file and check the dependencies' licenses +against it. -The verification is case-insensitive, and is done on both the `license` -and the `classifiers` metadata fields. See -https://pypi.org/project/liccheck/[liccheck]'s documentation for more -details. +If you want to add a license to the list, feel free to open a Pull Request +against https://github.com/fedora-infra/shared[the shared repo]. -You can automate the license check with the following snippet in your -`tox.ini` file: +You can automate the license check by running the shell script in Tox. -.... -[testenv:licenses] -deps = - liccheck -commands = - liccheck -s .license_strategy.ini -.... - -Remember to add `licenses` to your Tox `envlist`. === Security @@ -271,3 +183,7 @@ commands = .... Remember to add `bandit` to your Tox `envlist`. + +Note that https://docs.astral.sh/ruff/[Ruff] is also capable of running +security checks equivalent to Bandit's. Make sure you are selecting the +`S` category in `pyproject.toml`.