Using pytest asyncio to Reduce Test Execution Time

Milos Kajkut

Posted On: November 30, 2023

8 Min

Longer test execution times can hinder development cycles, slow down continuous integration pipelines, and impede overall responsiveness in the software development process. Therefore, optimizing test execution time is critical in achieving streamlined, agile, and high-quality software development processes.

When discussing decreasing execution time for automated test cases, the first thought is parallel execution of test cases. But is a parallel execution the only solution for this problem or is the test execution bottleneck of the test automation framework? We should ask ourselves this question whenever we want to reduce test execution time.

Chapters

Overview

pytest-asyncio is a pytest plugin that runs asynchronous tests, allowing concurrent I/O operations and improving overall test execution speed for async Python code.

How to Use asyncio With pytest?

To implement asynchronous testing with pytest, follow these structured steps that allow concurrent execution of preconditions and efficient test management within async-based Python testing frameworks.

  • Create Async Functions: Define two asynchronous precondition functions that perform setup tasks concurrently. Use await statements to manage non-blocking execution and ensure tasks finish independently without sequential delays.
  • Initialize Fixture: Create an asynchronous fixture responsible for setup and teardown processes. Include precondition calls and use asyncio.gather() to run them simultaneously within the fixture context.
  • Run Preconditions Concurrently: Use asyncio.gather() to launch multiple coroutines together. This ensures both preconditions execute in parallel, saving time compared to running each task sequentially.
  • Implement Async Tests: Write asynchronous test functions that consume the prepared fixture. These functions should await necessary operations and verify outputs after the preconditions complete successfully.
  • Use Asyncio Marker: Apply @pytest.mark.asyncio to enable async test execution. This decorator ensures pytest recognizes the test as asynchronous and handles event loops properly.
  • Enable Auto Mode: Configure asyncio_mode=auto in pytest.ini to simplify async handling. It removes the need for manually adding the @pytest.mark.asyncio decorator.
  • Execute Tests: Run pytest normally through the command line or IDE. The asynchronous coroutines execute concurrently, producing faster and more efficient test execution results overall.
  • Analyze Results: Review test logs to confirm that precondition functions run simultaneously. You’ll observe improved runtime performance and reduced total execution time due to concurrent execution.

TABLE OF CONTENTS

Methods to Reduce Test Execution Time

When it comes to managing and optimizing test execution time in pytest, two main approaches are prominent: the classical setup and teardown mechanism and harnessing the potential of asynchronous programming using asyncio.

  • Using Setup and Teardown mechanism
  • Using asyncio mechanism

First, let’s talk about automated test cases. Each automated test case contains three main parts: Setup, Actual Test Case, and Teardown. In the Setup process, we perform all necessary actions to prepare the environment for test execution. Those actions include reading or writing data from a database, performing API requests, reading or writing data from specific files, etc. After the process, we can start with the Test Case execution process, where test steps and assertions will be executed. As a final stage, the process of Teardown will be executed to revert all changes made during the setup and test execution process.

Setup and Teardown in pytest

Setup and Teardown process can be performed by Fixtures. Fixtures in pytest are a very powerful feature, and by definition, they present a Python function that pytest executes before and after the actual test function. So, let’s look at how to make a Fixture:

Fixture implementation is fully flexible (which can be concluded from the example above), but the right question is: “Where is the before and after part in this fixture?”. For the setup and teardown process, we need to implement the Fixture as a generator function using the yield keyword. So, everything before the yield keyword will be considered as setup, and everything after the yield keyword will be considered a teardown. This is how it looks in our example:

After implementing the Fixture, we are ready to write the first test case:

So far, we have implemented one Fixture with setup and teardown process and one test case that uses the implemented Fixture. From this point, we can set up a problem that can be solved with pytest asyncio. Imagine a situation where you implemented 200 automated test cases and ran them in parallel, but still, test execution takes too long.

The first step is to detect bottlenecks in the framework or to give an answer to the question, “Where do we have time leakage? ”. For this purpose, we will set up the problem situation like this:

We will add one more test case and modify the test case with time.sleep() to simulate test case execution time:

Also, to simulate precondition functions, we will add to the function that time execution will also be simulated with time.sleep():

At the end, let’s modify the Fixture to execute precondition functions:

From the execution report, we can see the following results:

As a starting point, we should detect and count bottlenecks in the framework:

  • 1st Bottleneck: Assume 2 seconds for test execution is the best time we can achieve with the current test implementation. For the execution of two test cases, we need 4 seconds.
  • 2nd Bottleneck: Assume that 3 seconds for precondition execution is the best time we can achieve with the current implementation. For execution precondition functions, we need 6 seconds per test case.

We have set up the problem situation and can discuss possible solutions for the bottlenecks. The solution for the first bottleneck is obvious: we need to introduce parallel execution, and we will reduce test time execution from 4 seconds to 2 seconds. But, with parallel execution, we will not solve the second bottleneck, and there are an additional 6 seconds in total time execution.

Asyncio Module in Python

Asyncio is a powerful Python module that enables developers to write asynchronous code by leveraging the intuitive async/await syntax. This module is especially valuable when dealing with I/O-bound operations, including network requests, file operations, and other tasks requiring external resources. Introduced in Python 3.4, the asyncio module has emerged as an integral component of Python’s robust asynchronous programming capabilities. It provides a structured and consistent approach to asynchronous programming using coroutines, the event loop, and other related concepts.

How to use asyncio with pytest?

Our goal with pytest asyncio is to run two precondition functions concurrently. To achieve such a goal, it is necessary to create coroutines from precondition functions:

To run coroutines concurrently, we need to use await asyncio.gather:

At the end, we need to make test functions also coroutines. To do that, pytest provides two possible solutions:

  • The first solution is to mark test functions with @pytest.mark.asyncio and use the keyword async:

  • The second solution is to set up the parameter asyncio_mode=auto in the pytest.ini file. In that case, we do not need to decorate the test function with marker @pytest.mark.asyncio:

Now, see the results:

From the results, we see that by the concurrent approach, we reduced execution time by 3 seconds and, in total, 10 seconds.

Conclusion

In conclusion, we delved into two powerful strategies, each tailored to optimize the efficiency of test execution in pytest. The approach of setup and teardown provides a structured means of preparing the testing environment before execution and cleaning up afterward, ensuring a consistent and controlled context for each test. Adopting asynchronous programming through pytest asyncio introduces a dynamic paradigm, particularly advantageous for handling I/O-bound tasks and achieving concurrent execution.

By exploring both these approaches, you can balance meticulous test preparation and swift execution, ultimately reducing test execution time and enhancing the overall effectiveness of the testing process. Whether leveraging traditional setup practices or embracing the asynchronous capabilities of asyncio, pytest offers a robust framework for optimizing test workflows in diverse testing scenarios.

Milos Kajkut

Milos Kajkut is a test automation engineer for Web and Mobile Qt (Android and iOS platform) applications. Selenium with Python is used for Web test automation. Squish and Appium with Python are used for Mobile test automation.

See author's profile

Author

Milos Kajkut is a test automation engineer for Web and Mobile Qt (Android and iOS platform) applications. Selenium with Python is used for Web test automation. Squish and Appium with Python are used for Mobile test automation.

Blogs: 1

linkedintwitter