How Continuous Testing Can Improve DevOps Efficiency

What is DevOps?

The best way for businesses to accelerate the pace, efficiency, and quality of their development pipeline is through DevOps. DevOps has an outstanding track record because it significantly changes how engineering teams work together to develop, test, and produce software. DevOps is broad and inclusive, encompassing not only the use of new frameworks and development practices but also a complete philosophy and way of thinking that motivates various implementations throughout our industry.

From Wikipedia:

“DevOps is a methodology in the software development and IT industry. Used as a set of practices and tools, DevOps integrates and automates the work of software development (Dev) and IT operations (Ops) as a means for improving and shortening the systems development life cycle. DevOps is complementary to agile software development; several DevOps aspects came from the agile way of working.”

The formal definition as you can see focuses on the technical aspects and methodology, but in my opinion, DevOps brings more to the table. It is a completely new culture of innovation, ownership, and continuous learning activities with the main goal of improving the software development life cycle right from definition, and development to release.

It is not a surprise that DevOps has become the standard these days. The benefits it provides are for all to see:

• Acceleration of the development process.
• Quality is built-in starting from the earliest stages of development.
• Improves collaboration and ownership between engineering and operations.
• Reduces development and business roadblocks.
• Accelerates engineering productivity.
• Optimizes the software testing process so that it can give an effective result in less time.

With that being said, keep in mind that DevOps is like Agile. It is NOT a silver bullet, but it can (and should) have a great impact on your organization and customer experience.

The main benefits of Continuous testing

• Faster time-to-market: DevOps practices enable organizations to deliver software updates and new features faster and more frequently, reducing the time-to-market.
• Increased collaboration and communication: DevOps fosters a culture of collaboration and communication between development and operations teams, resulting in more efficient and effective software development and deployment.
• Improved quality: By integrating automated testing and continuous integration and delivery (CI/CD) pipelines, DevOps helps ensure that software is thoroughly tested and of high quality before deployment.
• Greater agility and flexibility: DevOps practices enable organizations to respond quickly to changing market conditions and customer needs, and to adapt their software development and deployment processes accordingly.
• Increased efficiency and cost savings: DevOps reduces waste and inefficiencies in the software development process, resulting in cost savings for organizations.
• Better reliability and stability: By implementing practices such as monitoring and logging, DevOps helps organizations identify and resolve issues quickly, resulting in more reliable and stable software.

Overall, DevOps is an approach that enables organizations to deliver high-quality software faster and more efficiently, while also fostering a culture of collaboration and continuous improvement.

Key Challenges

Continuous testing is essential for businesses to stay one step ahead of their competition, but it also comes with inherent complexities and challenges that are worth mentioning before you dive right in.

1. Automating everything – Prior to continuous testing becoming the new norm in cloud-based developments (SaaS products), organizations’ primary objectives were to cut back on testing efforts, particularly manual testing. Therefore, the primary question that persisted throughout was, “Are we automating enough?” In the past, this was an effective strategy, but in the era of continuous testing, it needs to be reevaluated. Massive unstructured test suites that are the result of the “Automating Everything” effort become unmanageable in a matter of months and take far longer to maintain than the testing itself. Furthermore, organizations and their development teams cannot run them continuously as part of a structured process per fix per build which embraces the main motivation of automating main components in the SDLC. What has changed in the modern era, then? We must act quickly and concentrate on the tests that will yield the highest return on investment. This is a result of realizing that automating everything is just not practical.
2. Inadequate utilization of resources – Organizations must make investments in resources, infrastructures, frameworks, and manual labor if they want a Continuous testing process that is effective and scalable. If it is not managed and tracked over time, this could be very expensive. For instance, a daily CD process utilizes a variety of servers (both physical and virtual), each of which consumes resources that, when added up over time, can be quite expensive. (This is relevant even more if you use cloud provider such as Azure, AWS and more).
    
   Engineering teams will be expected to ensure that they are not wasting these valuable resources on pointless job executions that fail due to inadequate coding and testing. This can be a major issue for startups or low-budget projects that cannot afford to waste money on unutilized resources. As a result, it is critical to keep a constant eye on the CD execution pipeline to identify and address areas where resources are being wasted.
3. Continuous testing is not just for testing – The idea that continuous testing is solely focused on testing is one of the most common misconceptions. This is definitely not the case. Even though testing is at its core, there are a number of other areas involved in the process that we must concentrate on in order to make the most of continuous testing.

For example:

• Creating health dashboards related to each test suite.
• Executing test suites on multiple versions, platforms, and frameworks.
• The ability to re-run tests in case of failure (While keeping the failures logs, dumps and any other information relevant for debugging).
• Generating auto reports to relevant stakeholders.

Thoughts on DevOps and Testing

DevOps involves many different parts like tools, infrastructure, coding standards, monitoring, and many more, that depend on one another to function smoothly. But, let’s now concentrate on the element that is most intriguing to us: testing.

Although DevOps has significant benefits, you cannot get them without the right implementation. One example that I see all the time is the neglect of quality to increase speed. Therefore, you must find the correct balance between speed and quality, if you prioritize one thing over the other you will find very soon that the field and customer experience will suffer substantially and may have a great deal of impact on the business reputation. So, before focusing on speed, remember that quality cannot be neglected. Bear in mind that one of the significant core pillars of DevOps is to optimize the test process and activities so it can become more effective.

So, what can we do to increase the scale of quality?

1. Do your best to incorporate testers into the DevOps ecosystem. Testing should not exist outside but it should be an integral part of it.
2. Shift your testers into DevOps teams to increase the focus on quality and not just on speed and process.
3. Let them focus on exploratory test sessions before integrating a new feature into the core repository.
4. Let your testers focus on quality risks and how to remove them.
5. Ensure that defect management is in sync with the test analysis results of the CI/CD pipeline (finding defects early in the development lifecycle and preventing possible defects from emerging later in the production cycle is the main goal).

Concluding Thoughts

Simply put, the main objective of continuous testing is to build quality into the product from the very beginning of the software development life cycle. Using this method, we push our teams to make culture, mindset and technical adjustments to make sure they can test at all stages of development (Unit testing, integration testing, Security Testing, system testing and acceptance testing) and accelerate the overall development process to achieve better results for their customers.

That being said, there is one problem that is worth mentioning here, what about the developers? How do they react to the new job requirements involved in the adaption of continuous testing? We have seen a significant improvement in the last decade in the “Programmer” mindset that tests are not part of the SDLC and are solely the responsibility of testers. This was an undeniable fact in traditional waterfall SDLC, and we saw how it began to shift in Agile frameworks, making it impossible for developers to not take a decisive part of test ownership . However, DevOps has made significant progress by embracing CI/CD processes and a continuous testing mindset.

From the standpoint of the developer, they must concentrate on coding and avoid spending their time on tasks like debugging and monitoring tests. In DevOps, developers must alter their perspective and accept the fact that testing is a crucial component of their new role. As engineering teams look to deliver software faster (mostly by automating all the manual labour involved in traditional development) the quality of their work still needs to be evaluated.Mistakes can be very costly and can impact the entire CI/CD process and we all know, a poor product delivery to the field, can have a decisive and permanent impact on the business reputation and lead to legal risks.

Continuous testing should be embraced right from the beginning, early in the development stage, to increase the feedback loops that will guarantee that risks are identified, mitigated and monitored. Another point worth mentioned is the “Quality ownership”. So, in traditional SDLC we have testers that took care of it, and in Agile frameworks (Scrum, Kanban etc.) the boundaries started to become vague meaning that developers took more responsibilities on quality. And in DevOps? If you embrace it right, there will no owners at all, as taking care on quality becomes everyone’s responsibility.

David Tzemach

The founder and owner of the Agile Quality Made Easy blog (25K followers). A platform that he uses for teaching and coaching others, sharing knowledge with people, and guiding them towards success while giving them the inspiration and tools to discover their own path.

See author's profile