Tox, Travis CI and Coveralls: my config

The goal

I recently updated a project of mine (django-minipub), I already had Travis CI and Coveralls set up, I wanted to add Tox to the mix and to get all 3 to play nicely together. I found many helpful tutorials online, but none fitted exactly my requirements. So here's my own implementation; please continue reading if you want to do exactly as follows:

Have Tox installed and running on your project.
Have Travis CI up and running - and have it reuse the build matrix already defined in the Tox configuration file. No duplication required!
After Travis has successfully completed - run Coveralls once (and only once). I've found lots of Python/Django projects where Coveralls was called for every combination of Python/Django under test, which seemed like a waste.

Why?

Tox allows contributors to an open source project to run the project's test suite against all the supported versions of Django and Python automatically and rapidly, giving them an opportunity to catch obscure bugs before submitting a Pull Request.
Travis CI does the same thing - but on a server, so that the information is available to the person (or people) who review and accept Pull Requests. Basically it helps prevent "but it worked on my machine!" issues, and also ensures that any new code will not break for some weird combination of Python and Django.
Coveralls measures test coverage for a project - it's a "nice to have", it's not a metric that should be interpreted too strictly, but is a useful tool for tracking if new code is not being run through any tests whatsoever.

Starting with tox

Here is an example tox configuration file:

# tox (https://tox.readthedocs.io/) is a tool for running tests
# in multiple virtualenvs. This configuration file will run the
# test suite on all supported python versions. To use it, "pip install tox"
# and then run "tox" from this directory.

[tox]
envlist =
    py{35,36}-django111
    py{35,36,37}-django22
    py{36,37,38}-django30

[testenv]
deps =
    django111: django>=1.11,<2.0
    django22: Django>=2.2,<2.3
    django30: Django>=3.0,<3.1
    -r{toxinidir}/example_project/requirements.txt
changedir =
    {toxinidir}/example_project/
commands =
    python manage.py test

Running quickly through the different sections:

envlist defines the various combinations of Python and Django that your code must support.
deps defines more precisely the Django versions that can be used - and also pip install s any required packages.
changedir - change the working directory to wherever the manage.py file is found.

Adding Travis

I will build the Travis configuration file in 2 phases.

The first version:

dist: xenial
sudo: false
language: python

# Travis config is driven by the tox.ini file.

# Here we just specify:
# 1. The Python versions to run - it will start a CI server for each, and run all the tox environments
#    that use that Python. So in Travis we are running both in parallel (one server per Python version)
#    and in series (each server runs all the Django envs for that version of Python).

python:
  - "3.5"
  - "3.6"
  - "3.7"
  - "3.8"

install:
  - pip install tox-travis

script:
  # Run against all the Python interpreter and Django versions specified in the tox.ini file.
  - tox

The key information is the use of the tox-travis package, which helps integrate Tox and Travis. I've seen lots of Django projects where the Tox configuration file specifies which combinations of Django and Python should be tested - and then the Travis configuration file repeats the same information, but in a different format!

Tox-travis allows Travis to read from the tox.ini file - so no more duplication.

After that, the only "duplication" is that in the Travis configuration file we still need to specify the Python versions to run. This is a "feature" that puzzled me at first. When running Travis, a new machine (I say "machine" as a a simplification and to avoid getting bogged in discussions at to whether it's a VM or a Docker instance or...) is started for every combination of Python and Django under test - and once the test run is complete, in the Travis interface you will see a line per "combination" with its output.

Tox-travis is a bit different in that it starts just one new machineper Python interpreter. So - for example - in the above configuration file, a machine will be started for Python 3.5 - and it will run the test suite against both Django 1.11 and Django 2.2. This changes the Travis CI interface slightly, as you will get one set of logs per Python version - and several different runs of unit tests in it. This maybe makes it less clear at a glance exactly which combinations of Python and Django have crashed. It's a tradeoff - a bit less clarity vs. an easier to maintain configuration.

Adding Coveralls

Finally... we can amend the Travis configuration file, so that it also runs coverage. Here's the second version of the same configuration file:

dist: xenial
sudo: false
language: python

# Travis config is driven by the tox.ini file.

# Here we just specify:
# 1. The Python versions to run - it will start a CI server for each, and run all the tox environments
#    that use that Python. So in Travis we are running both in parallel (one server per Python version)
#    and in series (each server runs all the Django envs for that version of Python).

python:
  - "3.5"
  - "3.6"
  - "3.7"
  - "3.8"

install:
  - pip install tox-travis
  - if [ "$TRAVIS_PYTHON_VERSION" = "3.6" ]; then pip install coveralls -r example_project/requirements.txt; fi

script:
  # Run against all the Python interpreter and Django versions specified in the tox.ini file.
  - tox
  # Coveralls only needs to run once - so run it against just one Python interpreter.
  - if [ "$TRAVIS_PYTHON_VERSION" = "3.6" ]; then cd example_project && coverage run manage.py test; fi

after_success:
  - if [ "$TRAVIS_PYTHON_VERSION" = "3.6" ]; then coveralls; fi

So what's changed? Well, as mentioned above, tox-travis will start up one machine per version of Python. We only need to run Coveralls once - so the trick is to randomnly select one Python version - and only run Coveralls on the machine that's running that version of Python. The extra lines are:

Install any extra requirements - if you glance back to the tox.ini file, it also installs requirements for every test run - but here, we are going to run code outside of the Tox test suite, so we manually install packages.
Run coverage at the same time as we run the test suite.
after_success is only run if the main test suite ran successfully - there is no need to generate a coverage report if tests are failing.

Conclusion

We've specified just once the different combinations of Django and Python that are to be tested; which is nice!