== Tests

Tests make development easier for both veteran project contributors and
newcomers alike. Most projects use the
https://docs.python.org/3.10/library/unittest.html[unittest] framework
for tests so you should familiarize yourself with this framework.

[NOTE]
====
Writing tests can be a great way to get involved with a project. It's an
opportunity to get familiar with the codebase and the code submission
and review process. Check the project's code coverage and write a test
for a piece of code missing coverage!
====

Patches should be accompanied by one or more tests to demonstrate
the feature or bugfix works. This makes the review process much easier
since it allows the reviewer to run your code with very little effort,
and it lets developers know when they break your code.

=== Test Organization

Having a standard test layout makes it easy to find tests. When adding
new tests, follow the following guidelines:

[arabic]
. Each module in the application should have a corresponding test
module. These modules should be organized in the test package to mirror
the package they test. That is, if the package contains the
`<package>/server/push.py` module, the test module should be in a module
called `<test_root>/server/test_push.py`.
. Within each test module, follow the
https://docs.python.org/3.10/library/unittest.html#organizing-test-code[unittest
code organization guidelines].
. Include documentation blocks for each test case that explain the goal
of the test.
. Avoid using mock unless absolutely necessary. It's easy to write tests
using mock that only assert that mock works as expected. When testing
code that makes HTTP requests, consider using
https://pypi.python.org/pypi/vcrpy[vcrpy].

[NOTE]
====
You may find projects that do not follow this test layout. In those
cases, consider re-organizing the tests to follow the layout described
here and follow the established conventions for that project until that
happens.
====

=== Test Runners

Projects should include a way to run the tests with ease locally and the
steps to run the tests should be documented. This should be the same way
the continuous integration (Jenkins, Zuul CI, etc.) tool runs the
tests.

There are many test runners available that can discover
https://docs.python.org/3.10/library/unittest.html[unittest] based tests.
These include:

* https://docs.python.org/3.10/library/unittest.html[unittest] itself via
`python -m unittest discover`
* http://docs.pytest.org/en/latest/contents.html[pytest]
* http://nose2.readthedocs.io/en/latest/[nose2]

Projects should choose whichever runner best suits them.

[NOTE]
====
You may find projects using the
https://nose.readthedocs.io/en/latest/[nose] test runner. nose is in
maintenance mode and, according to their documentation, will likely
cease without a new maintainer. They recommend using
https://docs.python.org/3.6/library/unittest.html[unittest],
http://docs.pytest.org/en/latest/contents.html[pytest], or
http://nose2.readthedocs.io/en/latest/[nose2].
====

[[tox-config]]
=== Tox

https://pypi.python.org/pypi/tox[Tox] is an easy way to run your
project's tests (using a Python test runner) using multiple Python
interpreters. It also allows you to define arbitrary test environments,
so it's an excellent place to run the code style tests and to ensure the
project's documentation builds without errors or warnings.

You can find an example in our CookieCutter template's
https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/tox.ini[tox.ini].
It runs the test suite in various versions of Python, reports the coverage,
runs the pre-commit checks, makes sure we're not depending on software with
forbidden licenses, and builds the documentation with the `warnings treated
as errors` Sphinx flag enabled.


=== Coverage

https://pypi.python.org/pypi/coverage/[coverage] is a good way to
collect test coverage statistics.
http://docs.pytest.org/en/latest/contents.html[pytest] has a
https://pypi.python.org/pypi/pytest-cov[pytest-cov] plugin that
integrates with https://pypi.python.org/pypi/coverage/[coverage].
https://pypi.org/project/diff-cover/[Diff-Cover] can be used to ensure
that all lines edited in a patch have coverage.

It's possible (and recommended) to have the test suite fail if the
coverage is below 100%. See the template's
https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/pyproject.toml#L98[pyproject.toml]
file for an example.

An alternative to having 100% coverage is to forbid new changes from bringing
the coverage down. This can be checked by `diff-cover` with the following
`tox.ini` snippet:

....
[tox]
envlist = pyXX...,diff-cover,...
# If the user is missing an interpreter, don't fail
skip_missing_interpreters = True

[testenv:diff-cover]
deps =
    diff-cover
commands =
    diff-cover coverage.xml --compare-branch=origin/develop --fail-under=100
....

You must then make sure that the `pytest` command line contains
`--cov-report xml` to produce the `coverage.xml` file that `diff-cover` will
analyze.

New projects should enforce 100% test coverage. Existing projects should
ensure test coverage does not drop to accept a pull request and should
increase the minimum test coverage until it is 100%.

[NOTE]
====
https://pypi.python.org/pypi/coverage/[coverage] has great
https://coverage.readthedocs.io/en/coverage-4.3.4/excluding.html[exclusion]
support, so you can exclude individual lines, conditional branches,
functions, classes, and whole source files from your coverage report. If
you have code that doesn't make sense to have tests for, you can exclude
it from your coverage report. Remember to leave a comment explaining why
it's excluded!
====

=== Licenses

The https://pypi.org/project/liccheck/[liccheck] checker can verify that
every dependency in your project has an acceptable license. The
dependencies are checked recursively.

The licenses are validated against a set of acceptable licenses that you
define in a file called `.license_strategy.ini` in your project
directory. To avoid having to maintain and sync the same file in every
project, a shared file is available
https://raw.githubusercontent.com/fedora-infra/shared/main/liccheck-strategy.ini[in the shared repo].

The CookieCutter template contains
https://github.com/fedora-infra/cookiecutter-python-app/blob/main/%7B%7B%20cookiecutter.slug%20%7D%7D/devel/run-liccheck.sh[a shell script]
that will download the shared file and check the dependencies' licenses
against it.

If you want to add a license to the list, feel free to open a Pull Request
against https://github.com/fedora-infra/shared[the shared repo].

You can automate the license check by running the shell script in Tox.


=== Security

The https://pypi.org/project/bandit/[bandit] checker is designed to find
common security issues in Python code.

You can add it to the tests run by Tox by adding the following snippet
to your `tox.ini` file:

....
[testenv:bandit]
deps = bandit
commands =
    bandit -r your_project/ -x your_project/tests/ -ll
....

Remember to add `bandit` to your Tox `envlist`.

Note that https://docs.astral.sh/ruff/[Ruff] is also capable of running
security checks equivalent to Bandit's. Make sure you are selecting the
`S` category in `pyproject.toml`.
