Flask testing basics

Software is everywhere, and a new one appears even as we speak. To create good software you not only need to understand the problem it solves. You also need to have enough quality checks before releasing it for usage as a product. The aspect of quality is where tests come in. Tests are written to make sure the software is bug-free and with no unexpected behavior with different inputs. The cost of fixing a bug in testing is usually very less compared to fixing it in production.

In this topic, we will identify what needs to be tested, explore different testing practices, understand some recommended strategies, and finally look at Pytest, a very popular testing library in Python. Software testing is both an art and a science, and we should understand that writing good tests is a skill that helps one write better software. Therefore, it is important to self-update on this as much as one can.

Identifying what to test

Testing is an opinionated subject and identifying what to test is different for different projects. There is a way of development called test-driven development (TDD) where you write tests first and then implement the feature. That is more suitable when you are implementing something new. But let's see some best practices for existing projects, like an existing e-commerce website:

• Start with testing the critical components of your system, things that cannot be out of service for the system to work, for example, a database that cannot be absent for your system to work

• Define how users will use your system, and define tests in accordance with that, for example, how should an item search be processed in the backend for a logged-out user vs a logged-in user vs a first-time user?

• For any new feature of the system, add tests. For example, how to enable a logged-out user to place orders in a system with minimum information.

Types of tests

There are several types of tests in different stages of a software project. Each has its own purpose. To understand better, let's look at their definitions and some examples:

When you start the project, you should focus on writing more unit tests to test smaller components of the project. Then as more features appear you can focus on writing more integration and functional tests to business-wise validate your work. And as the project becomes a real-life big project with periodic deployments, focus on regression tests to keep deployments smooth, performance tests to abate any performance degradation, and acceptance tests for final acceptance to the customer.

Pytest: getting started

Several testing libraries exist in Python like unittest, nose, doctest, and others. But pytest is the most commonly used library because of its simple setup and good features to write different types of tests. In order to install pytest, we need to run the following command

pip install pytest

Once you install pytest, it runs via the command line or via Pycharm IDE directly. To run it via Pycharm, select Preferences > Tools > Python Integrated Tools -> Default Test Runner -> pytest.

Best practices for pytest are as follows:

• Store all tests in a tests/ directory

• File names should strictly start with test_

• Function names should strictly start with test

Pytest example 101

Let's create a file util.py with the following code to test. This is a simple util function that checks if a number is even which raises TypeError in case the input is not an integer.

# util.py
def is_even(num):
    if not isinstance(num, int):
        raise TypeError("Invalid input type, only integers allowed")
    return num % 2 == 0

For creating tests on it we should create another file test_feature.py with the following code.

# test_feature.py
from .util import is_even
def test_even():
    assert is_even(2) == True

Now there are two ways to run pytest, one using a command line, where you can run the command pytest from the directory of util.py file. You would get the following output.

The second is using Pycharm where you can run the tests directly by right-clicking on the file and then see the test results.

Usually, pytest runs any file with a name starting with "test_", but this setting can be changed. Also, pytest relies on assert keyword in Python to evaluate a test as success/failure. If you write your tests as classes (much like the unittest framework in Python) then name your class and test methods containing the substring "test". For example,

class Testme:
    def test_method(self):
        assert is_even(2) == True

    def do_not_run_this_test(self):
        assert 1 == 2

Here the test_method will be executed, while the do_not_run_this_test will be skipped.

Testing scenarios with pytest

Pytest provided customization for running tests to make the test suite useful. You will see some of the use cases of testing and what support pytest provides. We are running all the following examples for the is_even function in the command line using pytest -rA -v command, -rA prints the detailed report and -v increases verbosity.

• Running the same test with multiple inputs

  • It may be needed to run the same test with multiple inputs to examine the code from multiple angles. pytest provides @pytest.mark.parametrize decorator to do this:

    @pytest.mark.parametrize("num, res", zip([0, 1, 2, 3, 4, 5], [True, False, True, False, True, False])) 
    def test_many(num, res): 
        assert is_even(num) is res
    

  • For tests marked as @pytest.mark.parametrize we see it is run with multiple inputs with the test_many function. The input parameters num, res are passed one by one and evaluated as can be seen in the logs. Also, note that the name of the variables defined in @pytest.mark.parameterize("num, res"... should match the input to the test_many(num, res).

  • 

• Handling exceptions inside tests

  • Sometimes we expect exceptions in tests, for example, if you pass a string input to the is_even function you get a TypeError in Python, as you cannot divide a string by 2. To handle this, pytest provides pytest.raises for custom exception handling.

  • def test_exception(): with pytest.raises(TypeError) as e: result = is_even("abc") assert e.value.args[0] == "Invalid input type, only integers allowed"

  • 

• Skipping tests based on the environment they are run

  • Some tests work only in certain environments, for example, you can have tests that run only in Python version 3 versus tests that run in Python version 4.

  • To run them all together you can use @pytest.mark.skipif decorator specifying conditions to skip the test.

  • Also, you can entirely skip a test using @pytest.mark.skip by providing a reason.

  • import sys @pytest.mark.skip(reason="The API is yet not developed for string inputs") def test_invalid_input(): assert is_even("xyz") is False @pytest.mark.skipif(sys.version_info.major != 4, reason="Run this test with python 2 only") def test_conditional_skipping_odd(): assert is_even(5) is False @pytest.mark.skipif(sys.version_info.major != 3, reason="Run this test with python 3 only") def test_conditional_skipping_even(): assert is_even(6) is True

  • 

• Running tests that share a common grouping logic

  • Sometimes it is desirable to run tests together that have a common theme of existence, or you can mark a group of tests to run together in order to test a specific scenario.

  • @pytest.mark.non_zero_integer_only def test_even(): assert is_even(2) is True @pytest.mark.non_zero_integer_only def test_odd(): assert is_even(3) is False def test_for_zero(): assert is_even(0) is True

  • Two tests, test_even and test_odd are marked in the same group. In order to run them you can use the command pytest -rA -v -m "non_zero_integer_only" , -m to specify the marker name. Note: without defining the marker in pytest.ini you will get a PytestUnknownMarkWarning. The output of this test is as follows. (Note only non_zero_integer_only marked tests are run)

  • 

Software Tests as Documentation

While the primary purpose of software tests is to ensure code correctness and reliability, they also serve as valuable documentation. Well-written tests act as living documentation for your codebase, providing clear examples of how different components should behave and interact. This can significantly enhance a project's maintainability and ease the onboarding process for new developers.

To maximize the documentative value of your tests, follow these best practices: use descriptive test names that clearly state what's being tested, include setup context to show necessary preconditions, write clear and specific assertions, comment on complex logic when needed, and group related tests logically. By treating tests as documentation, you ensure that your "documentation" remains current and accurate, as tests are typically updated alongside the code that is being tested.

Conclusion

You took a deep dive into understanding how testing works in general, what are the types of tests, and what are some of the best practices for testing. Then you took pytest and looked at how to run it in general and with markers. Pytest has many other features like fixtures, profiling which are useful in maintaining a strong test suite. You also explored how tests can serve as valuable documentation for a project, providing clear examples of component behavior and enhancing code maintainability. Testing is always a crucial domain to master and it should be used to write better software.