Best practices in software engineering

Input data for tests

As we saw in the last section, when using parametrisation it's often useful to split your test function into two logical parts:

  1. The data to be tested
  2. The code to do the test

This is because we had a situation where we had one test function and multiple examples to test. The opposite situation also happens where we have multiple test functions, all of which want the same input data.

The name that pytest uses for "data which are provided to test functions" is fixture since it fixes a set of data against which to test.

We'll start with the example of the add_arrays function to explain the syntax but soon we'll need to use a example which demonstates the benefits more.

To make things clearer, we'll trim down the test file back to the basics. Just one test for add_arrays:

test_arrays.py
from arrays import add_arrays

def test_add_arrays():
    a = [1, 2, 3]
    b = [4, 5, 6]
    expect = [5, 7, 9]
    
    output = add_arrays(a, b)
    
    assert output == expect

Both of these tests use the same values for a and b. In this example it's not really a big problem that we're repeating ourselves but in a more complex test suite the data we're testing over may be very complicated or slow to create.

To avoid repeating ourselves, we will create a fixture which will provide the data to our tests. To create our fixture we define a function which is decorated with the pytest.fixture decorator. Apart from that, all the function needs do is return the data we want to provide to our tests, in this case, the two input lists:

import pytest

@pytest.fixture
def pair_of_lists():
    return [1, 2, 3], [4, 5, 6]

To make the test functions make use of the fixture, we use the name of the fixture (pair_of_lists) as a parameter of the test function, similar to how we did with parametrisation:

def test_add_arrays(pair_of_lists):
    ...

The data are now available inside the function using that name and we can use it however we wish:

def test_add_arrays(pair_of_lists):
    a, b = pair_of_lists
    ...

This isn't how functions and arguments usually work in Python. pytest is doing something magic here and is matching up the names of things which it knows are fixtures (due to the decorator) with the names of parameters to test functions, automatically running the fixture and passing in the data.

Note that pair_of_lists here is not a test function. It does not contain any asserts and will not explicitly appear in the pytest output.

Putting it all together, we end up with:

test_arrays.py
import pytest

from arrays import add_arrays

@pytest.fixture
def pair_of_lists():
    return [1, 2, 3], [4, 5, 6]

def test_add_arrays(pair_of_lists):
    a, b = pair_of_lists
    expect = [5, 7, 9]
    
    output = add_arrays(a, b)
    
    assert output == expect

When we run the test suite, pytest will automatically run the pair_of_lists function for each test and pass in the result.

$
pytest -v
=================== test session starts ====================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /home/matt/projects/courses/software_engineering_best_practices/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/matt/projects/courses/software_engineering_best_practices
plugins: nbval-0.9.5
collected 10 items                                         

test_arrays.py::test_add_arrays PASSED               [ 10%]
test_books.py::test_word_counts[hat-33] PASSED       [ 20%]
test_books.py::test_word_counts[freedom-71] PASSED   [ 30%]
test_books.py::test_word_counts[electricity-1] PASSED [ 40%]
test_books.py::test_word_counts[testing-3] PASSED    [ 50%]
test_books.py::test_word_counts[Prince-1498] PASSED  [ 60%]
test_books.py::test_word_counts[internet-0] PASSED   [ 70%]
test_books.py::test_word_counts[Russia-71] PASSED    [ 80%]
test_books.py::test_word_counts[Pierre-1260] PASSED  [ 90%]
test_books.py::test_word_counts[None-566311] PASSED  [100%]

=================== 10 passed in 35.32s ====================

A different example

It might be hard to see the benefit of fixtures with this rather contrived example so lets take a look at a more sensible one where using a fixture makes sense.

Make a new file called books.py which contains the following:

books.py
def word_count(text, word=''):
    """
    Count the number of occurences of ``word`` in a string.
    If ``word`` is not set, count all words.
    
    Args:
        text (str): the text corpus to search through
        word (str): the word to count instances of

    Returns:
        int: the count of ``word`` in ``text``
    """
    if word:
        count = 0
        for text_word in text.split():
            if text_word == word:
                count += 1
        return count
    else:
        return len(text.split())

To test this function we want a corpus of text to test it on. For the purposes of this example and to simulate a complex data input, we will download the contents of a particularly long novel from Project Gutenberg. Our test function uses urllib.request to download the text, converts it to a string and passes that to the word_count function.

At first we will simply check that the word "hat" appears 33 times in the book:

test_books.py
import urllib.request

from books import word_count

def test_word_counts():
    url = "https://www.gutenberg.org/files/2600/2600-0.txt"
    book_text = urllib.request.urlopen(url).read().decode('utf-8')
    assert word_count(book_text, "hat") == 33
$
pytest -v test_books.py
=================== test session starts ====================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /home/matt/projects/courses/software_engineering_best_practices/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/matt/projects/courses/software_engineering_best_practices
plugins: nbval-0.9.5
collected 1 item                                           

test_books.py::test_word_counts PASSED               [100%]

==================== 1 passed in 2.59s =====================

The test has passed and it took about two seconds. This is because it takes some time to download the file from the internet. For this example we want it to take some time as it helps demonstrate the point. In reality you will come across test data inputs which take some time (more than a few milliseconds) to create.

This creates a tension between wanting to have a large test suite which covers your code from lots of different angles and being able to run it very quickly and easily. An ideal test suite will run as quickly as possible as it will encourage you to run it more often. It's a good idea to have at least a subset of your tests which run through in some number of seconds rather than hours.

Two seconds is not bad for this test but if we want to test against multiple examples, it could get slow. Let's parametrise the test to add in a bunch more inputs:

test_books.py
import urllib.request

import pytest

from books import word_count

@pytest.mark.parametrize('word, count',  [
    ('hat', 33),
    ('freedom', 71),
    ('electricity', 1),
    ('testing', 3),
    ('Prince', 1498),
    ('internet', 0),
    ('Russia', 71),
    ('Pierre', 1260),
    (None, 566311),
])
def test_word_counts(word, count):
    url = "https://www.gutenberg.org/files/2600/2600-0.txt"
    book_text = urllib.request.urlopen(url).read().decode('utf-8')
    assert word_count(book_text, word) == count
$
pytest -v test_books.py
=================== test session starts ====================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /home/matt/projects/courses/software_engineering_best_practices/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/matt/projects/courses/software_engineering_best_practices
plugins: nbval-0.9.5
collected 9 items                                          

test_books.py::test_word_counts[hat-33] PASSED       [ 11%]
test_books.py::test_word_counts[freedom-71] PASSED   [ 22%]
test_books.py::test_word_counts[electricity-1] PASSED [ 33%]
test_books.py::test_word_counts[testing-3] PASSED    [ 44%]
test_books.py::test_word_counts[Prince-1498] PASSED  [ 55%]
test_books.py::test_word_counts[internet-0] PASSED   [ 66%]
test_books.py::test_word_counts[Russia-71] PASSED    [ 77%]
test_books.py::test_word_counts[Pierre-1260] PASSED  [ 88%]
test_books.py::test_word_counts[None-566311] PASSED  [100%]

==================== 9 passed in 30.67s ====================

You see here that it took about nine times as long. This is because the file is downloaded afresh for every test example where really, it only needs to be downloaded once.

Let's move the slow setup into a fixture and give that as a parameter of the test function:

test_books.py
import urllib.request

import pytest

from books import word_count

@pytest.fixture()
def long_book():
    url = "https://www.gutenberg.org/files/2600/2600-0.txt"
    book_text = urllib.request.urlopen(url).read().decode('utf-8')
    return book_text

@pytest.mark.parametrize('word, count',  [
    ('hat', 33),
    ('freedom', 71),
    ('electricity', 1),
    ('testing', 3),
    ('Prince', 1498),
    ('internet', 0),
    ('Russia', 71),
    ('Pierre', 1260),
    (None, 566311),
])
def test_word_counts(long_book, word, count):
    assert word_count(long_book, word) == count
$
pytest -v test_books.py
=================== test session starts ====================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /home/matt/projects/courses/software_engineering_best_practices/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/matt/projects/courses/software_engineering_best_practices
plugins: nbval-0.9.5
collected 9 items                                          

test_books.py::test_word_counts[hat-33] PASSED       [ 11%]
test_books.py::test_word_counts[freedom-71] PASSED   [ 22%]
test_books.py::test_word_counts[electricity-1] PASSED [ 33%]
test_books.py::test_word_counts[testing-3] PASSED    [ 44%]
test_books.py::test_word_counts[Prince-1498] PASSED  [ 55%]
test_books.py::test_word_counts[internet-0] PASSED   [ 66%]
test_books.py::test_word_counts[Russia-71] PASSED    [ 77%]
test_books.py::test_word_counts[Pierre-1260] PASSED  [ 88%]
test_books.py::test_word_counts[None-566311] PASSED  [100%]

==================== 9 passed in 33.67s ====================

Perhaps surprisingly, it is still taking very long time!

By default a fixture will run once for every test function that uses it. In our case we only need it to run once for all the tests in the test session so we can pass in the scope parameter to pytest.fixture and set it to session:

test_books.py
import urllib.request

import pytest

from books import word_count

@pytest.fixture(scope="session")
def long_book():
    url = "https://www.gutenberg.org/files/2600/2600-0.txt"
    book_text = urllib.request.urlopen(url).read().decode('utf-8')
    return book_text

@pytest.mark.parametrize('word, count',  [
    ('hat', 33),
    ('freedom', 71),
    ('electricity', 1),
    ('testing', 3),
    ('Prince', 1498),
    ('internet', 0),
    ('Russia', 71),
    ('Pierre', 1260),
    (None, 566311),
])
def test_word_counts(long_book, word, count):
    assert word_count(long_book, word) == count
$
pytest -v test_books.py
=================== test session starts ====================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /home/matt/projects/courses/software_engineering_best_practices/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/matt/projects/courses/software_engineering_best_practices
plugins: nbval-0.9.5
collected 9 items                                          

test_books.py::test_word_counts[hat-33] PASSED       [ 11%]
test_books.py::test_word_counts[freedom-71] PASSED   [ 22%]
test_books.py::test_word_counts[electricity-1] PASSED [ 33%]
test_books.py::test_word_counts[testing-3] PASSED    [ 44%]
test_books.py::test_word_counts[Prince-1498] PASSED  [ 55%]
test_books.py::test_word_counts[internet-0] PASSED   [ 66%]
test_books.py::test_word_counts[Russia-71] PASSED    [ 77%]
test_books.py::test_word_counts[Pierre-1260] PASSED  [ 88%]
test_books.py::test_word_counts[None-566311] PASSED  [100%]

==================== 9 passed in 4.05s =====================

Now it only takes about as long as a single test did since the slow part is only being done once.

Exercise

Add some more parameters to the test and check that it doesn't take any longer to run