The Role of Continuous Testing in Delivering Quality Digital Experiences

We all want to deliver quality experiences to our customers. Before we can ensure a quality experience for our customers, it is essential to establish a clear definition of quality specific to the product we are developing. Once established, embracing continuous testing becomes crucial as it can increase the speed at which features are delivered to customers, enhancing the overall customer experience.

Quality is subjective. W. Edwards Deming wrote that “quality can only be defined in terms of the agent” [1]. The development team will need to understand quality from the point of view of each of the agents for the product they are working on. An example would be If you were to consider agents for a book. The first agent for a book is the writer, and for the writer, quality could be, considering whether they got their message across. Another agent would be a printer, and for the printer, quality would relate to the size of the print run, the quality of the paper and the quality of the ink. The publisher is another agent for a book. For the publisher, quality may be whether they sold a lot of books. The reader is another agent; for the reader, quality would be whether the book was a good read or contained some spark of information. The development team needs to work out who the agents are for their product, and what quality is for them. Some of the agents will be external, such as a manager at one of their customer’s offices, and some of the agents will be internal such as Customer Success. Once the team has identified the agents it can then develop a testing strategy to achieve a quality experience for each of the agents.

Shift-Left and Shift-Right Testing in Continuous Testing Strategy

Continuous testing includes shift-left and shift-right testing and it is a strategy that the team should consider using in order to deliver a good quality digital experience. Development can be visualized as a timeline where writing the specification is positioned on the left-hand side of the line and production is represented on the right-hand side of the line. Continuous testing encompasses both leftward and rightward shifts along the timeline. It involves conducting testing as early as possible in the development process (leftward shift) and also performing testing in the production environment after code deployment (rightward shift).

Testing as early as possible in the development process can be described as shift-left testing, however, traditionally, testing takes place once the code is ‘ready for testing’. Once the code has been tested and bugs have been fixed the code is deployed to production. Advocates of lean processes have argued that if testing is left until the code is ‘ready for testing’ then the product’s quality is already there and testing will have little impact on improving it. Lean aims to reduce waste and lean testing would mean starting to test as early as possible in the development process. Testing in this way reduces waste because if a bug is found in the specification, it is much cheaper to fix than a bug found in the code when the code is ‘ready to test’.

Shift-left testing, as a component of continuous testing, plays a crucial role in delivering quality digital experiences. By conducting testing activities earlier in the development process, shift-left testing enables the identification of bugs and issues at an early stage. As a result, it reduces the likelihood of encountering last-minute bugs just before the release. It will be less likely that the product will be delivered late because bugs will have been found early in the process. Bugs that are found early in the process are cheaper to fix because, for example, it is cheaper to fix a fault in a specification than it is to fix a bug in production. There will be higher customer satisfaction because the code is delivered on time. If testing starts as early as possible and continues throughout the development process there will not be a testing phase and this will help deliver features earlier.

There are many types of testing that can be done as a part of shift left testing. The specification can be tested before development has started. One such type is testing the specification itself before the actual development begins. This critical examination of the specification aims to identify any errors or inconsistencies. An effective approach to testing the specification is through a collaborative “three amigos” session, where a tester, developer, and product owner work together to review the specification from their respective perspectives. This collaborative effort helps ensure a comprehensive and well-rounded evaluation of the specification, minimizing the chances of errors and enhancing overall quality.

Implementing Continuous Integration, Automated Testing, and Code Quality Enhancement

Implementing a Continuous Integration (CI) system is crucial for the development team to ensure the smooth building, testing, and integration of the product with every code merge. When a developer’s code passes the automated tests in CI gives the developer reassurance that their code changes have not broken the product.

An automated test strategy should be created by the development team. The strategy should include unit testing, API testing, and integration testing. The automated tests should be executed in CI.

Linting and static analysis are important ways of improving code quality. Developers should use linters and static analysis locally when they are developing the code because they will find issues in the code as it is being written. Additionally, the CI system should incorporate linting processes, providing the team with an added layer of confidence in the codebase. Linting in both local development and the CI system helps ensure that coding standards are followed, potential bugs are caught early, and the overall code quality is improved.

Incorporating Shift-Left and Shift-Right Testing for Enhanced Quality and User Experience

As part of shift-left testing, the tester should execute tests to ensure that the new feature being developed delivers a quality experience for each user or stakeholder of the product. These tests should cover various scenarios and validate that the feature meets the requirements and expectations of different agents, such as end-users, administrators, or any other relevant parties. This further helps the team to proactively address any issues, improve the feature’s quality, and enhance the overall user experience.

The availability of various types of shift-left testing provides a development team with a valuable opportunity to incorporate quality into the product from the early stages. By leveraging different testing techniques, such as unit testing, integration testing, security testing, etc, the team can identify and address potential issues and vulnerabilities before they escalate.

While shift-left testing moves testing earlier in the development process, shift-right testing moves testing later in the development process. With shift-right testing, we test to the right on the timeline in production. Testing in production enables us to learn how the application is performing and enables you to find any production issues quickly. We can gain information that enables the team to know what errors are occurring in production by monitoring the application for errors. It is useful to run non-functional tests in production, such as performance tests, to find what performance issues customers are experiencing. The team will also want to perform security testing to check that the application is secure.

Shift-right testing helps to deliver quality digital experiences in several ways. It reduces the risk of releasing software because faults in the software are discovered quickly. This means that new features and fixes can be deployed to production more quickly. Test coverage can be expanded by automating tests that run regularly in production to cover user test cases. Different ways of deploying code can be used such as a/b testing in order to gain information from customers about how best the product can be designed.

As a part of shift-right testing, the tester should consider the needs of every agent involved in the product. needs of each of the agents for the product and design automated tests that can be run in production to test whether a quality experience is being delivered for that agent.

Unlocking Code Confidence with Continuous Testing

The development team will gain confidence in their code when continuous testing is adopted as a testing strategy. This is because the code is tested from the earliest stage of development right through to production. This confidence can enable the development team to release their code more quickly. This can be done by using continuous delivery which means that the code and infrastructure are kept in “a deployable state, and that all code checked into trunk can be safely be deployed to production”[2]. It can also be done using continuous deployment, which means that in addition to continuous delivery, developers are “deploying good builds into production on a regular basis through self-service” [3]. Releasing code more quickly improves the quality of the customer’s experience because the customers can benefit from new features and fixes more quickly.

Some definitions of continuous testing define continuous testing as only covering automated testing. If this is the case then a number of testing activities that would give the development team confidence in their code are excluded. It would help the team if it includes code reviews, exploratory testing and considered testability.

Code reviews should be included in shift-left testing as they are a powerful way of sharing knowledge across the team and finding issues with the code.

The development team can do exploratory testing on feature branches, or by using feature flags. This will enable the development team to explore and test the new features as these are being developed.

The team ought to also consider how testable the code is, and endeavor to make it testable. If the code is hard to test it will be hard to deliver a quality experience for the customer. An example of how to make the code more testable would be to improve error logging.

There is a lot of discussion about continuous testing. The adjective ‘continuous’ is added to testing to show that testing is happening all the time. Continuous means that testing is happening without a break, replicating the behavior typically associated with the machines Testing, though, is a human activity that involves analyzing the test results, gaining insights and applying this knowledge to inform the next piece of work. For instance, continuous integration and continuous deployment have mechanical aspects and emphasize the seamless and uninterrupted nature of these specific processes. These processes will also include running automated tests.

However, testing should contain breaks when the testers and development team wait for test results or analyze them. Further, testing will also contain breaks when the development team and testers are learning something related to their development activities. Testing is better described as ‘continual’ rather than continuous because continual describes a process where there are plateaus and periodic breaks in testing to facilitate learning. It is better to adopt a strategy of continual testing, rather than continuous testing because continual testing will include breaks in testing when the team is studying the results of tests and learning.

The adjective continual is widely used to describe an activity that requires pauses and breaks. Last year, I was fortunate to have a conversation with Kevin Cahill, who is W. Edwards Deming’s grandson. While talking to him I used the phrase continuous learning. He told me that he had once said the words continuous learning while talking to his grandfather. His grandfather told him that he should say continual learning and that he should look up the definitions of continuous and continual in the dictionary. Since then I have noticed that books such as The DevOps Handbook contain sections about continual learning and experimentation[4]. DevOps owes a great deal to Deming [5] and continuous testing relates to DevOps. Testing is better described as continual rather than continuous because it is done by humans we need the plateaus and breaks to reflect and learn.

Continual testing should use a plan-do-study-act-cycle because it is a way of working that facilitates learning. This means that once a piece of testing is planned and implemented, the next step is to analyze the test results and incorporate the insights gained into the next testing phase. If the team works in this way, they will continually be learning from their testing, and their testing will improve.

Conclusion

The role of continuous testing is a valuable testing strategy that helps to deliver quality digital experiences because it enables quicker delivery of features to customers through continuous delivery and continuous deployment. This is because it finds issues with the software early in development, and gives confidence in the development process. Continual testing has the same benefits as continuous testing and it also gives the development team space to learn from their work.

References
[1] Out of the Crisis by W. Edwards Deming (1986, p168)
[2] The DevOps Handbook by Gene Kim, Jez Humble, Patrick Debois and John Willis (2016, p5, p37 & p176)
[3] ED – Rediscovering a WWII Statistician& Why We Need His Ideas to Survive the Digital Age by John Willis