How Agile Teams Use Test Pyramid for Automation?

Product testing is considered a very important step before the product is released to the end customer. Depending on the nature and complexity of the project/product, you need to make sure that you use the very best of testing methodologies (manual testing, smoke testing, UI testing, automation testing, etc.) in order to unearth bugs and improve product quality with each release.

‘Automation testing strategy’ has a lot of benefits over manual testing since automation test frameworks such as Selenium can save a lot of time. It also helps in lowering the overall costs and improving the test coverage.

The ROI (in terms of issues raised or time saved) of automation testing depends a lot on the planning & execution of the testing. As a test lead, you have to come up with an automation testing strategy depending on the project requirements and discussing with the relevant stakeholders of the project.

By analyzing the scope, size, and deadlines of the project; leads of the project have to take a conscious decision whether to use traditional waterfall or modern agile approach for testing.

Many times, team leaders plan the execution of the automation testing task at the end of the development cycle since the product is in ‘mature shape’ and the essential elements/features (UI, services, apps, etc.) of the product are more or less developed.

This approach may not drive the best results from automation testing. A better approach would be to have a mix of automation tests with regression tests at an early stage of product development. In fact, many organizations are now adapting the idea of Shift-left testing where you aim to test right from the very start of your client requirement gathering phase.

Mike Cohn’s concept of ‘Agile Test Pyramid’ can be instrumental in building a sound strategy for implementing automated software testing. ‘Agile test pyramid’ is one of the ideal mechanisms to signify the difference between traditional software testing and testing for ‘iterative development’.

When your organization is switching to agile approach for projects, your project management team should look at the factors that can contribute to Agile success since it requires a ‘mindset change’ at all the levels within the team (and the organization).

Before we have a look at the Agile test automation Pyramid, we should first have a look at the traditional model that is used for software testing.

If you’re new to Selenium and wondering what it is then we recommend checking out our guide – What is Selenium?

Introduction To The Traditional Testing Model

This model is primarily used for finding bugs in the software. When a team follows the ‘waterfall model’ for development & testing, the testing is performed manually through the User Interface (UI). As shown in the figure below, automation testing is done in the middle layer & the lower layer. The middle layer consists of the services that expose Application Programming Interfaces (APIs) that can be used by the UI layer. Unit tests constitute the lowest layer of the inverted pyramid and these tests are written by the developers in order to verify the ‘features coded by the developers’.

This model can be used in projects where the overall budget is low, requirements are clear, and fewer changes are expected during later phases of product/project development. The major disadvantage of this model is that it cannot be used for projects that have a higher budget. Also, where project deadlines are strict since the testing only starts after the completion of ‘product development’.

Since the testing activity starts at a later point in time; finding bugs & getting them resolved by the developers may take more time and resources. This would negatively impact the overall turnaround time which might delay shipping of the project/product. In such scenarios, the ‘Agile approach’ is used in today’s fast changing organizations due to its iterative & flexible approach.

As tests performed at the user-interface layer increase the duration of ‘test, feedback & fix’ cycle, majority of the organizations are switching to Test pyramid which uses minimized user-interface based test automation.

Cons Of Using The Traditional Testing Model

In the traditional model of testing, the primary focus of the testers developing the test code was developing UI centric test cases. Even the low-level functionalities that involve databases transactions, access to the core business logic, etc. was done via the UI test cases.

This approach is brittle in nature as the automation tests would be required to change if the web-application or website undergoes major UI changes. Hence, this involves constant maintenance costs which can increase the overall budget of the project. This could also impact the ‘project delivery timelines’ which may result in causing losses to the organization (as well as the client).

The other problem with this approach is that the automation test development was being done by the test team/testers that work in their own comfort zone and rarely get an opportunity to interact/discuss their issues/suggestions with the development team. Hence, there is minimal opportunity to build domain competency within the test team as there is less engagement across different teams (i.e. development, project planning, project management, testing, etc.).

All these problems can be solved if your test team adapts the agile test pyramid for devising their automation testing strategy. We have a closer look at the core building blocks of Agile test Pyramid in subsequent section.

Detailed Look At Agile Test Pyramid

The ‘Agile Test Pyramid’ was first proposed by Mike Cohn in his book ‘Succeeding With Agile’. Below is a simplistic view of the ‘Agile Test Pyramid’

From the figure, you can figure out that the ‘agile test pyramid’ is nothing but an inverted ‘ice cream cone’ as seen in traditional testing approach. Unlike that approach, where the ice cream cone can melt if the base is weaker, in agile the base of the pyramid is stronger since agile testing relies more on automation. Since there is more contribution from the developer team working on the project, there is increased engagement between the teams. The core agenda of the testing activity also shifts from ‘finding bugs’ (Traditional testing) to ‘prevention of bugs’ (Agile approach to testing).

Adapting to this approach also requires a shift in the mindset at every stage within the organization. In Agile methodology, complicated user-requirements are divided into number of ‘user stories’. The whole idea behind creating a user-story is to look at a requirement by wearing the shoes of the customer and then look for ways to achieve it in a simplistic manner.

User Stories can contain details about a feature, technical/non-technical requirement, business requirement, etc. Experienced & efficient agile teams can expedite the entire testing process by minimizing the time taken to convert acceptance criteria for user-stories to automation tests.

Layers Of Agile Test Pyramid

Unit Test

Unit Tests form the base-layer of the Pyramid. Unit tests are normally written by the developers who write test code to verify the functionalities that were developed by them. As the test focuses on testing a particular feature/functionality, a good developer can come up with good number of unit test cases in a very short span of time

Unlike other types of tests, the unit-test is also written to discover bugs (at a unit or block level), but the information that a developer can derive from unit-tests is more when compared to other types of tests e.g. The developer can get information about a particular side-effect that his patch can have on other features (which are dependent on that fix) or origin of the issue or details about ‘where in the code’ the fault/issue had occurred. Hence, the turn-around time of unit-tests is very fast since the developer himself can verify the ‘fix/developed feature’ and fix the issues that are found during unit-testing

All the other types of functional testing like Integration testing, System testing, Sanity testing, etc. focus on testing the complete product. Hence, the time is taken to ‘validate product->report an issue-> assign to the right developer->fix the issue->re-validate the fix (along with other boundary tests)’ can be quite high. The possible shortcoming of unit testing is that the testing is limited to a ‘feature/requirement’, which is why you might not be able to locate system-level issues.

Normally unit tests are coded by the developer himself, but it is recommended that you also involve a tester (who has coding knowledge & detailed knowledge about the feature being developed) in the unit-test development so that a fresh & neutral perspective can be brought on the table of unit-testing

Some of the common tools that are used for unit-testing are ‘xUnit variants for development languages’ e.g. JUnit for cross browser testing using Selenium automation, xUnit.net for .Net, etc. This layer has around 50~60% contribution to the overall performance of ‘Agile Test Pyramid’. As the weightage of unit level automation is so high, it is recommended that teams spend good amount of time on developing quality unit tests.

Service/API Layer Tests

We discussed earlier the drawbacks of focusing too much on user-interface tests and most of those drawbacks can be removed if the ‘focus is shifted to the Service/API layer’. You can also term this layer as the ‘Middleware Layer’.

Most of the modern applications have support for ‘third party developers’ to build an ecosystem around the product or application. This is possible by exposing Application Programming Interfaces (API’s)/web-service for the outside world/third party developers. These APIs have ‘minimal/no’ dependency on the UI layer. Testers can make use of these APIs to come up with tests can be used to verify the underlying features of the product. Though the scope of the API testing might vary based on the type of the project/application, majority of the APIs focus on business logic, interactions with the database, etc.

The core logic that executes when you perform some activity at the UI layer would happen in this layer e.g. Updating an entry in a table (of a database) after ‘Submit’ button is clicked. Such a design makes the UI layer more independent from the core-logic of the product, limiting it only to send/receive messages

Since you are interacting with any UI widgets, the speed with which these tests can be developed & executed significantly great when compared to UI-focused tests. This makes the overall test-design less brittle in nature, as any change in the business logic can be quickly translated to equivalent test code. In case you are working on a hardware product, this middleware layer is used to keep the core features/functionalities hardware-agnostic i.e. any change in the underlying hardware will have minimal/no impact on the product design.

Some of the common tools that are used for ‘Service layer automation’ are Cucumber, Robot framework, Fitnesse, Concordion, Behat, etc. This layer has around 20~40% contribution to the overall performance of ‘Agile Test Pyramid’. A design philosophy change would be required for achieving the best results out of automation testing in the middle layer. Since most of the consumer-focused products being built today are developed by keeping the ‘third party developer-ecosystem’ in mind, software architects & developers already use to ‘The Humble Dialog Box pattern’ to get the most out of this layer.

User-Interface (UI) Tests

This is the layer where you would be testing the product more from an end-user’s perspective. As a tester, your top priority would be to ensure that ‘UI Design Flow’ is in-line with the design requirements. There is a high possibility that your product would have a ‘Service/Middle layer’ in which case, you would have already verified the ‘functionalities from a product perspective’ (without taking the user-interface into account).

This test is done to make sure that interaction through the UI widgets delivers the right output on the screen e.g. Once the ‘Submit’ button is pressed on the ‘New user sign-up’ page, the user should be prompted to sign-in after verifying his account. As mentioned earlier, UI level testing can take a lot of time since it has to interact with multiple blocks within the product (along with major/minor updates on the user screen i.e. UI).

Since the testing at UI level is so brittle, it is recommended to focus on these tests to only verify ‘UI flow & interactions’ without looking into the system functionality. We have already discussed the cons of user-interface testing in the earlier section titled Cons of using the ‘Traditional Testing Model’. Unless & until your product does not have a middle-layer or you have no other alternative to verify the business logic, it is ideal to keep the core focus of this test to ‘User Interface’ as any change in UX/UI would like ‘re-inventing the wheel from testing perspective’.

This layer has around 0~10% contribution to the overall performance of ‘Agile Test Pyramid’. Tools & frameworks like Selenium, HP QTP (Quick Test Professional), Web Application Testing In Ruby (Watir), etc. can be used to perform UI testing.

In today’s times, the User Interface layer is more than a layer that provides a mode of interaction with the end-user. It can also contain business logic and frameworks like Test ’em can be used to execute tests within the browser (and server calls are mimicked). Hence in such kind of scenarios, User-Interface tests can also be seen under End-To-End testing.

Advantages Of Agile Test Pyramid

Now that you are aware of the blocks that constitute the Pyramid, let’s have a look at some of the high-impact advantages of this model.

• Unlike the traditional approach to testing, testers using this Agile approach do not work in a silo and have a complete view about the product. They get an opportunity to interact with other key stakeholders of the project. There is more team-bonding as far as this approach is concerned. As the automation test-cases are implemented in co-ordination with the development team providing better automation code quality, & code coverage can be much better when compared to the traditional testing approach.
• ‘Code Coverage’ or ‘Automation Coverage’ is a crucial Key Performance Indicator (KPI) for any developer working on test code. When the test code is written by putting more emphasis on the user-interface layer, there is less probability of verifying the core business-logic or back-end functionality of the product. As the turn-around time (TAT) of testing via UI is more, you can expect lesser ‘coverage even via automation’. This is completely contrasted to what you can achieve by deploying the agile test pyramid model for your testing activity.
• Project requirements can change from time to time, something that is possible even during later stages of a project. Along with the code used for development, test code also requires timely maintenance. The cost of code maintenance in traditional model can be significantly higher as compared to Agile model. In case of Agile approach, user-requirements are broken down into different user-stories and a user-story is not considered to be ‘complete/over’ till the time, all the effective test cases (automation & manual, new & old) are working. The overall maintenance is much better in the Agile approach and there is better ROI as well.

LambdaTest – Achieving The Best Out Of Agile Test Pyramid

Even though you use the Agile model for testing, you may want to optimize some jobs (related to testing) so that you can make the best out of the resources in hand. If you are building a web product, you would have faced the problem where your code has to be tested against a ‘browser version’ which is not latest or some version of Operating System. Building a test infrastructure which can fulfill these requirements would involve huge investment.

Using a cloud based cross browser testing tool can prove to be efficient in such scenarios. As a Product Owner or Scrum Master, your job would also be to optimize the time taken to bring each user-story to completion. By utilizing LambdaTest’s Automation features, the test team can reduce the overall build & execution times to run UI based tests. The same can also be used to optimize the time spent on the activities planned at each layer of the Agile Automation Pyramid i.e. UI testing, Service layer testing, and Unit testing.

Once you sign-up for an account on LambdaTest, you can make use of the Real-time testing, Visual UI testing, and automated cross browser testing features to accelerate testing activity planned in each user-story.

Let’s have a closer look at some of the features that can accelerate your cross-browser testing activity

Automated Cross Browser Testing – Though User Interface testing carries less weight age as far as Agile Test Pyramid is concerned, making use of Automation Testing in LambdaTest can free up the resources for other activities. You can use the capabilities of the Selenium Grid for cross browser testing in co-ordinance with Python/C#/Java/PHP/etc. to come up with different browser compatibility testing scenarios.

The implementation which you have written using Local WebDriver interface can be ported to Remote WebDriver interface with minimal effort. You can also generate different capabilities using the LambdaTest Capabilities Generator. Below is a screenshot of the Selenium where tests using different capabilities (Chrome + Windows 10, Firefox 64.0 + Windows 10, etc.) have been executed

You can make use of ‘Automation Logs’ and ‘Analytics’ to debug problems that are discovered during the process of automation testing. Concurrent test execution/ Parallel testing in Selenium should be used in cases where you plan to speed up the entire test cycle.

Visual UI Testing – This feature can be used when you want to perform UI testing on different combinations of devices, browsers, and operating systems. As mentioned earlier, UI testing can be brittle & would require significant amount of time & effort for planning & execution. Three different variations of Visual UI testing are Screenshot, Responsive, and Smart Testing.

• Using Screenshot option, you can capture automated bulk screenshots over a variety of browsers. This would help you keep a daily basis track of the UI flow in different screens in your product. Since the Screenshot is saved on the cloud, it is easy to compare the changes in UI over a period of time. More information is available here.
• Responsive testing can be used for verifying the product features on different viewports (screen resolution) and screen sizes. As you are not looking at the ‘performance’ of the web-app/website at this point, using Responsive testing will be handy when you want to test your product for responsive screen testing across devices. Here’s more information on Responsive Test.
• Manual verification of UI screens can be a daunting task and is also prone to human errors. By making use of the smart visual regression testing feature on LambdaTest, keeping track of changes in UI (along with UI rendering issues) is simplified using ‘smart image comparison’. The automated bulk screenshots captured using Screenshot testing along with ‘Smart Testing’ is a powerful combination to help you pace up cross browser testing under the agile test pyramid

Conclusion

With the fast changing business scenario, time to market is very critical and any delay in project deadlines could result in significant amount of losses to the organization. Instead of the traditional testing model, Agile testing can be used to increase the overall productivity, reduce time-to-market, & improve the overall quality of the end product.

As the goal is to cut costs & improve quality, using the ‘Agile Test Pyramid’ can be instrumental in achieving the best results from every layer of the Pyramid (Unit testing, API testing, UI testing). The productivity & turn-around-time can be further improved by making use of a cross-browser testing platform like LambdaTest which can minimize the time taken for Visual UI tests, automation tests, etc. It is recommended that you carefully chart out each user-story when you plan to implement agile approach to testing.