Python Testing Using pytest, Tox, Travis-CI and Coveralls

pytest: Micro Testing The Code

pytest is a unit test framework that actually tests the nuts and bolts of each piece of your code. You'll need to actually write test cases and pytest will collect and run those tests for you.

I favor pytest over Python's default unittest framework because of it's simplicity. Furthermore, there are many plugins exist that check various aspects of the project, such as doctest integration, pep8 format checking, etc.

pytest has excellent documentation. Here is the particular configuration for the WlTrace project.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# setup.cfg

[tool:pytest]
addopts = -v -x --doctest-modules --ignore=setup.py --cov=wltrace --pep8
pep8maxlinelength = 80
pep8ignore =
    E402    # module level import not at top of file
    E241    # multiple spaces after ','
    E226    # missing white space around arithmetic operator
    E222    # multiple spaces after operator
    docs/source/conf.py ALL

The addopts option specific the arguments to call pytest:

• -v: verbose output. This shows what is being tested, instead of just a dot.
• -x: stop on first failed test
• --doctest-modules: integrate doctest
• --ignore=setup.py: do not do doctest on the top level setup.py
• --cov=wltrace: Test the coverage of the wltrace module, which is the top module of the WlTrace package. Requires pytest-cov plugin and the Coverage.py package.
• --pep8: check pep8 format compliance. Requires pytest-pep8 plugin.

Now we can use a single pytest command to fire up the tests and also get code coverage information during the test stored at .coverage file, which will be used later for Coveralls. This file should be ignored by your VCS.

Tox: Macro Testing The Environment

Tox is a tool to test your final package in various Python environments, such as Python 2, Python 3, PyPy, Jython, etc. It is mainly used to make sure the setup.py is configured properly and your final distribution package can be successfully installed under those Python environments. Furthermore, it issues the test command after installation to test if the project also run properly in those environments.

Here's the project's tox.ini:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

[tox]
envlist = py27,pypy

[testenv]
commands = pytest
deps =
    pytest-pep8
    pytest-cov
    pytest
    coverage

[testenv:pypy]
install_command = pip install git+https://bitbucket.org/pypy/numpy.git {packages}

Here we specified two Python environments: Python 2.7 and PyPy. Note the special install_command configuration for PyPy to deal with the Numpy package.

Travis CI: Continuous Integration

Travis CI is a online service to automatically build and test your project for continuous Integration, so it is easy to pinpoint to the single commit that breaks the build.

Here is the .travis.yml configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

language: python
python:
    - "2.7"
    - "pypy"
install:
    - |
        if [[ "${TRAVIS_PYTHON_VERSION}" = pypy ]]; then
            git clone https://github.com/yyuu/pyenv.git ~/.pyenv
            PYENV_ROOT="$HOME/.pyenv"
            PATH="$PYENV_ROOT/bin:$PATH"
            eval "$(pyenv init -)"
            pyenv install pypy-5.4.1
            pyenv global pypy-5.4.1
            pip install git+https://bitbucket.org/pypy/numpy.git
        fi
    - pip install tox-travis coveralls
script:
    - tox

after_success:
    - coveralls

Here we first install a recent version PyPy and also tox-travis to make tox play nice with Travis CI. We also install the coveralls tool to be used later to publish the coverage information to http://coveralls.io.

Note that we install numpy in pypy environment explicitly. This is because the tox-travis plugin has some difficulty utilizing the install_command configuration.

Coverage.py: Code Coverage

Coverage.py is a tool to measure the code coverage of Python programs. In the context of testing, it can be used to measure the code coverage when running pytest, which effectively translate code coverage to test coverage.

Previously we use the pytest-cov plugin to call this tool during test so that we translate the code coverage to test coverage. The coverage information is stored in the .coverage file.

Coveralls: Showcase Coverage

Coveralls is a web service that displays the code coverage information generated by the Coverage.py tool.

We installed coveralls package in Travis. The coveralls command will publish the coverage information to http://coveralls.io.