What is Continuous Testing?

Introduction

Testing is a crucial part of the Software Development LifeCycle(SDLC). Testing should be included in every stage of the SDLC to get faster feedback and bake the quality within the product. Test Automation can get you excellent results if it is implemented and used in an efficient way and Continuous Testing is the right approach.

The Continuous Testing market is expected to grow at a Compound Annual Growth Rate of 15.9% during the forecast period of 2018-2023 and reach $2.41 billion by 2023. 2017 was considered the base year for estimating the market size. [Source]

In this article we will discuss what Continuous Testing is, how to implement and get benefit out of it.

What is Continuous Testing?

Continuous Testing helps in providing faster feedback in all the stages of the Software Development Lifecycle(SDLC). In most of the cases in SDLC, it is seen that minimal automated tests are written at the core level hence increasing the pressure on the top level of the test pyramid to perform manual exploratory testing.

This actually hits the quality as catching an error after development is complete, is very costly. The below table shows the cost to fix a bug at Google and you can just witness it costs a whopping $5000 when a bug is discovered in the System Testing phase.

Continuous testing helps us to evaluate this fear of software getting failed by providing early feedback as soon as the code is committed to the repository. The main goal of Continuous Testing is to test early at all stages of the SDLC with automation, test as often as possible, and get faster feedback on the builds.

You might be knowing about the Go/No-Go meetings which are set up before every release, this meeting helps you in finding out whether you are headed in the right direction. It helps you to make a decision whether you are good to release the application with the respective features in the production or not.
Similarly, Continuous Testing works, it provides you the test results based on which you can decide to move to the next stage of development.

Using Continuous Testing, we can fix all the failures as soon as they occur before moving on to the next stage which eventually helps in saving time and money.

Why is Continuous Testing needed?

In one of my previous projects we were working on a mobile application to be developed for iOS as well as Android platforms. The client wanted everything to be automated from the inception itself. Any bug leakage into production meant it will impact the business directly and cost millions of dollars.

We were asked to present a plan for automation where testing would be carried out in every stage of development to minimize the risk of bug leakage. Hence, we decided to implement the test pyramid and create a CI/CD pipeline where testing would be done continuously at every stage.

Here is the graphical representation of the CI/CD Pipeline which can also be taken as a practical guide to implementing Continuous Testing in a project.

To bake the quality within the product, we came up with a plan to introduce testing at every stage in the pipeline, and as soon as any red flag appears, it should be fixed before we move on to another phase. So, as soon as the dev commits the code to the remote repository the following scans would run

1. Static Code Analysis: This will ensure that the best coding practices are followed and would alert us with code smells in case of any errors.
2. SecOps Scan: This will scan the code and all the libraries used within the code for any Security Vulnerabilities and raise a red flag in case of Critical, High, Medium or Low level vulnerabilities which should be taken care of.

Once the above scans are successful, the pipeline would move ahead and run the following tests in the Development Environment:

1. Unit Tests
2. Integration Tests
3. System Tests
4. End to End Tests

All of the above tests will ensure that the code is perfectly working fine as expected.

In case any of the above tests fails,the pipeline will break and a red flag will be raised. It will be the responsibility of the developer who committed the code to fix those respective failing tests. It’s not about playing the blame game, but finding the commit which broke the build and fixing it. The team will offer help to the developer to fix the issue.

After all the above-mentioned tests pass successfully, the build will be deployed to QA Environment where again End-to-End automated tests will run on the QA build as a part of Regression tests.

Once the End-to-End automated tests pass on QA build, QA will pick up the build and perform Manual Exploratory tests to uncover any further defects in the build. Once QA signs off the build, it will eventually be deployed to the UAT env where further rounds of testing will be done by the UAT team of testers and finally after the sign off, the build will be deployed into production.

This plan worked for us tremendously as we uncovered many issues in the first stage and the second stage of the pipeline itself when the Unit and Integration tests were run.

Static Code Analysis and SecOps scan helped us in implementing the best coding practices and using fixing vulnerable libraries by updating to the latest version or discarding and using the libraries which were less prone to vulnerabilities and also frequently updating them so the code is less prone to security risks.

Though we also discovered issues in the Exploratory testing which was done manually, however those were not that critical, most of the issues were resolved in the initial phase itself which provided us faster feedback.

Continuous Testing is the need of the hour and it is no longer Good but must to have in the SDLC Life Cycle

From my experience the following points are derived which states why Continuous Testing is needed:

• Requirements change frequently: With the requirement changing frequently, the need arises to change the code as well, and with every change we do, there is a risk involved.
  There are two risks involved here, whether the changed code will work as expected and another one is if this change impacts the existing code. With Continuous Testing, we can tackle both of these risks by setting up an automated pipeline which will run the unit, integration and eventually automated regression tests.
• Continuous Integration: With agile development in place, Continuous Integration has gained a lot of popularity where developers merge their code to the main branch as often as possible, to make it production ready. Continuous Testing helps us here as, before the merging takes place the code goes through a pipeline where automated tests are run against the build. If there is a failure, the code does not merge and a red flag is raised.
• To be Production Ready: With Continuous Testing, we can be production ready as all our checks and tests run on an automated pipeline as soon as the developer commits the code.
• Reduce Human Errors: In the case of regression tests, if an automated test is written it can serve as a documentation proof for the feature and help out to reduce human errors in testing.

Advantages of Continuous Testing

• Fast Feedback: In the traditional Software Development process, the team had to wait for the tester’s feedback, who would test the build manually after the developer had completed writing the feature from their end. After the tester’s feedback, they had to rework to fix the issues which were time-consuming and more costly. With Continuous Testing, we can get faster feedback on the newly committed code and save time and money.
• Quality baked within the product: With all the tests running in the automated pipeline, from Unit, Integration, Functional, Security, Performance, and end-to-end user journeys, we can be sure that quality is baked within the product itself and need not worry about releasing it to production.
• Reduces bug leakage: Continuous testing helps in eliminating the chances of bugs occurring in the build by providing us with timely updates about software failures.
• Minimize the Risks: It also helps to find the risk, address them and improve the quality of the product.

Important types of Continuous Testing

1. Unit Tests: This involves testing a piece of code in isolation. Basically testing every method written for the feature. The main objective of this test is to check that the code is working as expected, meaning that all the functionalities, inputs, outputs, and performance of the code are as desired.
2. Integration Tests: This involves testing the two modules together. The goal of this test is to check that the integration between the two components is working fine.
3. Regression Tests: This is the most widely used test and it is used to check that the existing functionality of the application is working as expected after the latest addition or modification to the code repository,
4. End to End Journey tests: These tests are added to check the end-to-end working of the software. The goal of these tests is to check that the end user is able to use the application end to end.

Future of Continuous Testing

With the ever-increasing demand for high-quality software and the economies flourishing with digitalization at its core, continuous testing is considered an important aspect.
A software company is required to respond to frequent changes happening daily in the SDLC and continuous testing is the answer.

The following points are the benefits of continuous testing:

1. To adapt to the frequent changes in the Software.
2. To achieve maximized automation in the delivery cycle and avoid loopholes in the process.
   Minimize human errors.
3. To provide a cost-effective Solution to the end customer.
4. Beat the competition and outperform the competitors by releasing bug-free Software.
5. Baking the quality within the product.

As the technology is progressing so is the need to upgrade the process as well. With continuous testing best results can be achieved.

Role of Cloud Services Platform in Continuous Testing

Last year I was working on a mobile application development project which had iOS as well as Android versions to be rolled out. Now, as it was planned to be rolled out in different regions around Germany, we studied mobile phone usage in the different areas of Germany and came to know that both Android and iPhones are used in all areas.

Hence, we concluded that we will need a combination of at least 6 devices(3 Android devices and 3 iPhones) to test our build. 2 devices to test minimum supported versions, 2 devices with the latest versions, and 2 devices with random versions between the highest and minimum supported ones.

The hard part was to get these devices, as mobile phones and their versions are updated nearly every 2-3 months so even if the organization invested and bought these devices, it would be required to update them occasionally as and when the new version launches. Here, the cloud platform services came to the rescue and we simply purchased the right plan as per our requirement and got access to the required Real Devices which helped us test the applications smoothly by running automated as well as manual exploratory tests on the Real Devices on the cloud platform.

In today’s fast-paced world, there are multiple platforms on which software works, from browsers to mobile phones and tablets. As we release the application to production, we need to make sure that it runs on all the desired platforms as expected and fix the things we find which are not working.

However, to do that, we need to test it on the respective devices/browsers to be sure that it works hassle-free. This is possible but will cost money and time as we will have to purchase the hardware and also put up the required resources to make it work. From hiring engineers to setting up the infrastructure.

As we are testing continuously, performing parallel runs on the different browsers and their respective versions OR on different mobile devices with different OS versions, these services help us test continuously by providing us with the required device, Browsers/OS and their variety of versions, so we catch the bugs early and using the early feedback fix the required issue and make stop the bug leakage.

Conclusion

Quality is a crucial part of the software and it needs to be baked in the software. Continuous Testing helps us with building the right product by implementing testing at every phase of the Software Development Life Cycle.

We need to be production ready with every feature we build, it is necessary to get fast feedback with a fail-fast strategy. There are various testing types available that help us implement continuous testing using automated pipelines.

Cloud Services Platforms provide us with the right infrastructure to keep up the pace with testing continuously by providing the required infrastructure.

Faisal Khatri

Faisal is a Software Testing Professional having 14+ years of experience in automation as well as manual testing. He is a QA, freelancer, blogger and open source contributor.

He loves learning new tools and technologies and sharing his experience by writing blogs.

See author's profile