An Introduction to Python Unit Testing with unittest and pytest

July 26, 2022

In this article, we’ll look at what software testing is, and why you should care about it. We’ll learn how to design unit tests and how to write Python unit tests. In particular, we’ll look at two of the most used unit testing frameworks in Python, unittest and pytest.

Introduction to Software Testing

Software testing is the process of examining the behavior of a software product to evaluate and verify that it’s coherent with the specifications. Software products can have thousands of lines of code, and hundreds of components that work together. If a single line doesn’t work properly, the bug can propagate and cause other errors. So, to be sure that a program acts as it’s supposed to, it has to be tested.

Since modern software can be quite complicated, there are multiple levels of testing that evaluate different aspects of correctness. As stated by the ISTQB Certified Test Foundation Level syllabus, there are four levels of software testing:

Unit testing, which tests specific lines of code
Integration testing, which tests the integration between many units
System testing, which tests the entire system
Acceptance testing, which checks the compliance with business goals

In this article, we’ll talk about unit testing, but before we dig deep into that, I’d like to introduce an important principle in software testing.

Testing shows the presence of defects, not their absence.

— ISTQB CTFL Syllabus 2018

In other words, even if all the tests you run don’t show any failure, this doesn’t prove that your software system is bug-free, or that another test case won’t find a defect in the behavior of your software.

What is Unit Testing?

This is the first level of testing, also called component testing. In this part, the single software components are tested. Depending on the programming language, the software unit might be a class, a function, or a method. For example, if you have a Java class called ArithmeticOperations that has multiply and divide methods, unit tests for the ArithmeticOperations class will need to test both the correct behavior of the multiply and divide methods.

Unit tests are usually performed by software testers. To run unit tests, software testers (or developers) need access to the source code, because the source code itself is the object under test. For this reason, this approach to software testing that tests the source code directly is called white-box testing.

You might be wondering why you should worry about software testing, and whether it’s worth it or not. In the next section, we’ll analyze the motivation behind testing your software system.

Why you should do unit testing

The main advantage of software testing is that it improves software quality. Software quality is crucial, especially in a world where software handles a wide variety of our everyday activities. Improving the quality of the software is still too vague a goal. Let’s try to specify better what we mean by quality of software. According to the ISO/IEC Standard 9126-1 ISO 9126, software quality includes these factors:

reliability
functionality
efficiency
usability
maintainability
portability

If you own a company, software testing is an activity that you should consider carefully, because it can have an impact on your business. For example, in May 2022, Tesla recalled 130,000 cars due to an issue in vehicles’ infotainment systems. This issue was then fixed with a software update distributed “over the air”. These failures cost time and money to the company, and they also caused problems for the customers, because they couldn’t use their cars for a while. Testing software indeed costs money, but it’s also true that companies can save millions in technical support.

Unit testing focuses on checking whether or not the software is behaving correctly, which means checking that the mapping between the inputs and the outputs are all done correctly. Being a low-level testing activity, unit testing helps in the early identification of bugs so that they aren’t propagated to higher levels of the software system.

Other advantages of unit testing include:

Simplifying integration: by ensuring that all the components work well individually, it’s easier to solve integration problems.
Minimizing code regression: with a good amount of test cases, if some modifications to the source code in the future will cause problems, it’s easier to locate the issue.
Providing documentation: by testing the correct mapping between input and output, unit tests provide documentation on how the method or class under test works.

Designing a Test Strategy

Let’s now look at how to design a testing strategy.

Definition of test scope

Before starting to plan a test strategy, there’s an important question to answer. What parts of your software system do you want to test?

This is a crucial question, because exhaustive testing is impossible. For this reason, you can’t test every possible input and output, but you should prioritize your tests based on the risks involved.

Many factors need to be taken into account when defining your test scope:

Risk: what business consequences would there be if a bug were to affect this component?
Time: how soon do you want your software product to be ready? Do you have a deadline?
Budget: how much money are you willing to invest in the testing activity?

Once you define the testing scope, which specifies what you should test and what you shouldn’t test, you’re ready to talk about the qualities that a good unit test should have.

Qualities of a unit test

Fast. Unit tests are mostly executed automatically, which means they must be fast. Slow unit tests are more likely to be skipped by developers because they don’t provide instant feedback.
Isolated. Unit tests are standalone by definition. They test the individual unit of code, and they don’t depend on anything external (like a file or a network resource).
Repeatable. Unit tests are executed repeatedly, and the result must be consistent over time.
Reliable. Unit tests will fail only if there’s a bug in the system under test. The environment or the order of execution of the tests shouldn’t matter.
Named properly. The name of the test should provide relevant information about the test itself.

There’s one last step missing before diving deep into unit testing in Python. How do we organize our tests to make them clean and easy to read? We use a pattern called Arrange, Act and Assert (AAA).

The AAA pattern

The Arrange, Act and Assert pattern is a common strategy used to write and organize unit tests. It works in the following way:

During the Arrange phase, all the objects and variables needed for the test are set.
Next, during the Act phase, the function/method/class under test is called.
In the end, during the Assert phase, we verify the outcome of the test.

This strategy provides a clean approach to organizing unit tests by separating all the main parts of a test: setup, execution and verification. Plus, unit tests are easier to read, because they all follow the same structure.

Unit Testing in Python: unittest or pytest?

We’ll now talk about two different unit testing frameworks in Python. The two frameworks are unittest and pytest.

Introduction to unittest

The Python standard library includes the unittest unit testing framework. This framework is inspired by JUnit, which is a unit testing framework in Java.

As stated in the official documentation, unittest supports a few important concepts that we will mention in this article:

test case, which is the single unit of testing
test suite, which is a group of test cases that are executed together
test runner, which is the component that will handle the execution and the result of all the test cases

unittest has its way to write tests. In particular, we need to:

write our tests as methods of a class that subclasses unittest.TestCase
use special assertion methods

Since unittest is already installed, we’re ready to write our first unit test!

Writing unit tests using unittest

Let’s say that we have the BankAccount class:

import unittest

class BankAccount:
  def __init__(self, id):
    self.id = id
    self.balance = 0

  def withdraw(self, amount):
    if self.balance >= amount:
      self.balance -= amount
      return True
    return False

  def deposit(self, amount):
    self.balance += amount
    return True

We can’t withdraw more money than the deposit availability, so let’s test that this scenario is handled correctly by our source code.

In the same Python file, we can add the following code:

class TestBankOperations(unittest.TestCase):
    def test_insufficient_deposit(self):
      # Arrange
      a = BankAccount(1)
      a.deposit(100)
      # Act
      outcome = a.withdraw(200)
      # Assert
      self.assertFalse(outcome)

We’re creating a class called TestBankOperations that’s a subclass of unittest.TestCase. In this way, we’re creating a new test case.

Inside this class, we define a single test function with a method that starts with test. This is important, because every test method must start with the word test.

We expect this test method to return False, which means that the operation failed. To assert the result, we use a special assertion method called assertFalse().

We’re ready to execute the test. Let’s run this command on the command line:

python -m unittest example.py

Here, example.py is the name of the file containing all the source code. The output should look something like this:

.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

Good! This means that our test was successful. Let’s see now how the output looks when there’s a failure. We add a new test to the previous class. Let’s try to deposit a negative amount of money, which of course isn’t possible. Will our code handle this scenario?

This is our new test method:

  def test_negative_deposit(self):
    # Arrange
    a = BankAccount(1)
    # Act
    outcome = a.deposit(-100)
    # Assert
    self.assertFalse(outcome)

We can use the verbose mode of unittest to execute this test by putting the -v flag:

python -m unittest -v example.py

And the output is now different:

test_insufficient_deposit (example.TestBankOperations) ... ok
test_negative_deposit (example.TestBankOperations) ... FAIL

======================================================================
FAIL: test_negative_deposit (example.TestBankOperations)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "example.py", line 35, in test_negative_deposit
    self.assertFalse(outcome)
AssertionError: True is not false

----------------------------------------------------------------------
Ran 2 tests in 0.002s

FAILED (failures=1)

In this case, the verbose flag gives us more information. We know that the test_negative_deposit failed. In particular, the AssertionError tells us that the expected outcome was supposed to be false but True is not false, which means that the method returned True.

The unittest framework provides different assertion methods, based on our needs:

assertEqual(x,y), which tests whether x == y
assertRaises(exception_type), which checks if a specific exception is raised
assertIsNone(x), which tests if x is None
assertIn(x,y), which tests if x in y

Now that we have a basic understanding of how to write unit tests using the unittest framework, let’s have a look at the other Python framework called pytest.

Introduction to pytest

The pytest framework is a Python unit testing framework that has a few relevant features:

it allows complex testing using less code
it supports unittest test suites
it offers more than 800 external plugins

Since pytest isn’t installed by default, we have to install it first. Note that pytest requires Python 3.7+.

Installing pytest

Installing pytest is quite easy. You just have to run this command:

pip install -U pytest

Then check that everything has been installed correctly by typing this:

pytest --version

The output should look something like this:

pytest 7.1.2

Good! Let’s write the first test using pytest.

Writing unit tests using pytest

We’ll use the BankAccount class written before, and we’ll test the same methods as before. In this way, it’s easier to compare the effort needed to write tests using the two frameworks.

To test with pytest we need to:

Create a directory and put our test files inside it.
Write our tests in files whose names start with test_ or end with _test.py. pytest will look for those files in the current directory and its subdirectories.

So, we create a file called test_bank.py and we put it into a folder. This is what our first test function looks like:

def test_insufficient_deposit():
  # Arrange
  a = BankAccount(1)
  a.deposit(100)
  # Act
  outcome = a.withdraw(200)
  # Assert
  assert outcome == False

As you have noticed, the only thing that changed with respect to the unittest version is the assert section. Here we use plain Python assertion methods.

And now we can have a look at the test_bank.py file:

class BankAccount:
  def __init__(self, id):
    self.id = id
    self.balance = 0

  def withdraw(self, amount):
    if self.balance >= amount:
      self.balance -= amount
      return True
    return False

  def deposit(self, amount):
    self.balance += amount
    return True

def test_insufficient_deposit():
  # Arrange
  a = BankAccount(1)
  a.deposit(100)
  # Act
  outcome = a.withdraw(200)
  # Assert
  assert outcome == False

To run this test, let’s open a command prompt inside the folder where the test_bank.py file is located. Then, run this:

pytest

The output will be something like this:

======== test session starts ======== 
platform win32 -- Python 3.7.11, pytest-7.1.2, pluggy-0.13.1
rootdir: \folder
plugins: anyio-2.2.0
collected 1 item

test_bank.py .                                                                                                   [100%]

======== 1 passed in 0.02s ========

In this case, we can see how easy it is to write and execute a test. Also, we can see that we wrote less code compared to unittest. The result of the test is also quite easy to understand.

Let’s move on to see a failed test!

We use the second method we wrote before, which is called test_negative_deposit. We refactor the assert section, and this is the result:

def test_negative_deposit():
  # Arrange
  a = BankAccount(1)
  # Act
  outcome = a.deposit(-100)
  # Assert
  assert outcome == False

We run the test in the same way as before, and this should be the output:

======= test session starts =======
platform win32 -- Python 3.7.11, pytest-7.1.2, pluggy-0.13.1
rootdir: \folder
plugins: anyio-2.2.0
collected 2 items

test_bank.py .F                                                                                                  [100%]

======= FAILURES =======
_____________ test_negative_deposit _____________
    def test_negative_deposit():
      # Arrange
      a = BankAccount(1)
      # Act
      outcome = a.deposit(-100)
      # Assert
>     assert outcome == False
E     assert True == False

test_bank.py:32: AssertionError
======= short test summary info =======
FAILED test_bank.py::test_negative_deposit - assert True == False
======= 1 failed, 1 passed in 0.15s =======

By parsing the output, we can read collected 2 items, which means that two tests have been executed. Scrolling down, we can read that a failure occurred while testing the test_negative_deposit method. In particular, the error occurred when evaluating the assertion. Plus, the report also says that the value of the outcome variable is True, so this means that the deposit method contains an error.

Since pytest uses the default Python assertion keyword, we can compare any output we get with another variable that stores the expected outcome. All of this without using special assertion methods.

Conclusion

To wrap it up, in this article we covered the basics of software testing. We discovered why software testing is essential and why everyone should test their code. We talked about unit testing, and how to design and implement simple unit tests in Python.

We used two Python frameworks called unittest and pytest. Both have useful features, and they’re two of the most-used frameworks for Python unit testing.

In the end, we saw two basic test cases to give you an idea of how tests are written following the Arrange, Act and Assert pattern.

I hope I’ve convinced you of the importance of software testing. Choose a framework such as unittest or pytest, and start testing — because it’s worth the extra effort!

If you enjoyed this article, you might also find the following useful:

Visual Studio Code: End-to-End Editing and Debugging Tools for Web Developers