The Need For An Integrated Enterprise Test Execution Environment

“Organizations cannot make testing “cool,” let alone efficient, if the quality of the test environment is in question.”

Source-Gartner’s ‘3 Steps to Improve Test Data Management for Software Engineering’

Enterprises, today, are dealing with a huge problem when it comes to testing. Given their size and complexity, testing is often siloed and each team/unit creates its own ‘environment’ using different tools and following different processes. This makes organization-wide decision-making a huge problem and results in slower release velocity and increased developer friction and feedback time.

What goes into managing a test automation environment

Test Automation can’t be successful unless the right environments are available and managed regularly. Our study indicates that 20% of total bugs found in a release are categorized as environment issues which in turn impacts the test cycle and increases the project cost.
Test automation environment management requires constant upkeep in terms of correct hardware, the right OS with patches, and dependent/related software with correct versions. Test environment management is categorized into two areas/parts based on the type of activity that is being performed and there are also various activities to be performed in each category.

Part 1- Development machines for developing automation scripts. Automation scripts are developed and maintained by an automation engineer.

Part 2- Execution environment for automation script execution. The automation test suite for execution is identified and deployed in the execution environment by automation testers.

Now, let’s delve into development machines. These machines are used for building and maintaining automation scripts by automation engineers. The following environment activities are being done on these machines:

1. Initial hardware provisioning and upgrades in the future to meet automation requirements
2. Operating system and regular patch upgrades.
3. Tool/framework IDE installation and upgrades to a newer version as required
4. Automation software installation and upgrades to a newer version as required

In the case of open-source software, development machines are used for developing and maintaining test automation frameworks as well. Depending on the automation tool/framework, relevant software needs to be deployed. For an open-source framework with Python+Selenium, the latest Python version and Selenium drivers need to be deployed along with IDE for building the scripts.

In the case of a commercial tool, the relevant tool’s IDE has to be installed (Tosca commander, UFT IDE, Worksoft Certify, etc.,). The software versions have to be updated frequently. The required hardware configuration and OS version are identified initially to support building and maintaining the scripts. Any upgradation to a new version requires a re-look into hardware configuration as well. OS is required to be updated for patches and to the latest version to support tool/framework software. Development machines require managing hardware, OS, and dependent/related software, and the automation engineer is expected to have the right privileges to manage his/her environment.

Now, coming to the execution environment. The execution environment is used for executing the automation scripts. This environment is like a production system for the testing team. The test team will have read-only access to ensure that the environment is not compromised by installing additional software. The execution environment is normally owned by the infra/environment team in the organization. The following environment activities are being done on execution machines.

1. Initial hardware provisioning and upgrades in the future to meet new requirements
2. Operating system and patch upgrades
3. Run time software installation (No IDE to be installed). example in the case of .Net, only .Net runtime is to be installed.
4. Optional server and database installation and regular backup in case of commercial tools.

The hardware capacity and configuration are identified based on the number of scripts to be executed per day. The scripts can be executed in parallel and/or distributed fashion for achieving cycle time reduction. For software, in the case of an open source using Python+Selenium, a Selenium grid is required to be set up and the latest Selenium drivers have to be updated regularly. License management is not required but the right plugins are required for integrating with software configuration management, CI/CD pipeline, test management, test data management platform, third-party cloud device providers etc.

On the other hand, commercial tools would have dependencies that are bundled with its executable. For example, Tosca would require a .Net runtime environment which is installed before deploying software. Tosca server, database, and DEX are set up and maintained regularly as part of the environment management. Optionally a license server is also to be deployed and managed for commercial tools.

Test automation environments will not be able to meet enterprise needs unless integrated with enterprise platforms such as CI/CD, Software configuration management, Test Management, and Test Data Management platforms.

The right plugins and adapters have to be installed on development and execution environments for integrating with enterprise platforms.
It is indeed a tedious task to do all these day in and day out.

What can enterprises do to ease the pain?

Enterprise execution environment.

Enterprises should look at bringing the various tools and multiple environments into one main execution environment to streamline testing. This might require a deep dive into testing practices and tools, going through expenses, and adding required integration, that spans across the organization. However, it is a task that has to be undertaken if you want to run on in-house test execution platforms.

Else, enterprises should look to adopt/migrate to third-party platforms that come with intelligent test execution features and out-of-the-box integrations.

Enterprises should view their enterprise execution environment with a strategic lens. The end goal should be– how to reduce heavy lifting in terms of development machines and the execution environment mentioned above and instead get them to focus on actual testing. Enterprises need to give their testers and developers the power to be nimble and agile by removing all possible hindrances that distract them from testing.

To again quote from Gartner “Product teams run tests to make informed decisions about potential risks and to build confidence in the quality of their products. However, if the teams involved in testing don’t trust their test environments or the provisioned test data, testing is less valuable and test results are more likely to be met with skepticism. Overall, poor test environments and poor TDM (Test Data Management) practices reduce teams’ enthusiasm when it comes to testing activities.”

It is high time that companies put their enterprise execution environment at the center of SDLC and enable a seamless digital experience for their customers.

LambdaTest’s Enterprise Execution Environment (E3) removes the pain of maintaining multiple environments to meet varied organization execution environments requirements and instead replaces it with one unified LambdaTest environment. Our JITO – Just In Time Orchestration executes tests across any framework, tools, language, device, browser, or API, at blazing-fast speed, under a unified execution environment. Try now!

(A version of this blog post was first published on https://mohanbachu.wordpress.com/)