Testing in Production: A Detailed Guide

When most firms employed a waterfall development model, it was widely joked about in the industry that Google kept its products in beta forever. Google has been a pioneer in making the case for in-production testing. Traditionally, before a build could go live, a tester was responsible for testing all scenarios, both defined and extempore, in a testing environment. However, this concept is evolving on multiple fronts today. For example, the tester is no longer testing alone. Developers, designers, build engineers, other stakeholders, and end users, both inside and outside the product team, are testing the product and providing feedback.

Today’s test environment, a product under test, underlying technologies, and test configurations (systems, platforms, browsers, and so on) are all pretty complex. The services perspective is quite strong; the days of a local setup to test the product are long gone. Cloud environments provide the size and simplicity of use required for sophisticated interface testing, but the production environment provides unique testing possibilities that cannot be completely integrated before release. And the restrictions that the test team faces, such as time, budget, and the availability of specialized testers, are more visible than ever before. With all of these variables at work, testing in production is unavoidable. However, rather than viewing testing in production as an option imposed on teams, if the underlying benefits are carefully considered, this activity may substantially aid in improving product quality. So, what exactly does it mean to test in production, what are some of the methods for doing so, and how does it help?

What is Testing in Production?

“Testing in production” was a frequent joke among software programmers around a decade ago. It was a shorthand way of saying something you would never do… until certain tools started enticing you to do just that. If you Google “testing in production,” you’ll quickly get your fill of “testing in production” memes.

Testing in production refers to the process of evaluating new code and features directly in the live environment rather than in a separate staging area. This approach allows for real-time feedback and assessment of software performance under actual usage conditions, enhancing the reliability of continuous delivery. Others who provide input to the product team may include tech people, marketing teams, or analysts. Testing in production provides more realistic testing chances, enhances application visibility between the core product team and consumers, and promotes continuous development through continuous testing. Testing in production is a critical strategy to incorporate into your testing process in today’s cloud environment.

Of course, faults reported from the field have a negative connotation: this simply means the testing efforts were less efficient. An issue reported from the field is given top attention and resolved as quickly as possible. While faults that arise in production might still reflect negatively on the team’s efforts, a test in-production endeavor now has a broader reach. Not only are reactive actions done in response to user-reported concerns performed, but also proactive planned testing activities are undertaken before the product is formally released.

Bonus information – It is associated with Right-shifting activity

Shift right is the practice of shifting certain testing later in the DevOps process to production testing. Real-world deployments are used in production testing to evaluate and measure an application’s behavior and performance. A shift-left test method is one way DevOps teams may increase velocity. Shifting left moves most testing earlier in the DevOps pipeline, reducing the time it takes for new code to reach production and function successfully. However, while many types of tests, such as unit tests, may readily shift left, other classes of tests cannot be executed without deploying a portion or the entirety of a solution. Although deploying to a QA or staging service can imitate a comparable environment, it is not a complete substitute for the production environment.

What Does Not Represent Production Testing?

Let’s be clear about what testing in production is NOT before we go any further with this topic. Clarifying this is especially crucial when taking into account what goes on in a production setting and the effects of poor production testing procedures. Continuous deployments and rollbacks are not part of production testing. It is also not being careless with what you expose. You should not employ real production data to guarantee that everything works when testing in production. That is still the purpose of the testing process in test environments, staging environments, and everywhere else your program is tested before it goes into production.

Production testing in your real production system is all about studying, making improvements, and informing your choices with the most up-to-date and accurate data available – after the code has already gone through the regular deployment process. Serving production traffic and actual users allows you to collect user data, test software modifications, and obtain monitoring indicators that you may use to determine whether what you give is what you anticipated.

Do you want to learn about how to reduce production failures and improve software quality? Check out the TestuConf 2022 event on Reducing Production Failures by Improving Software Quality

Why production test?

Companies have always attempted to guarantee that the software they generate has been properly tested for defects in development, staging, and pre-production settings before it reaches customers in production. Early detection of bugs avoids customers from encountering mistakes, enhancing consumer trust and overall happiness with a brand and its goods.

It is, however, difficult to catch all defects in development and staging. Engineering and QA teams can spend a significant amount of time and effort developing unit tests, test suites, and test automation systems, attempting to simulate the production environment, or manually verifying user flows with mock user data and test cases to expose bugs, only to discover that a critical corner-case was overlooked. In the end, even after extensive testing throughout development, many consumers may encounter problematic software.

In many cases, completely simulating live, real-world systems in a test environment is impossible. With all of the dependencies inherent in current production systems, as well as the numerous potential edge cases, production testing has become an essential component of DevOps and software testing. Top software businesses like Google and Amazon frequently deploy new features to a subset of their traffic to gauge their impact.

Let’s look at some of the benefits of testing in production.

• What you see is what the customer sees – In terms of production environments, there is no room for mistakes. You are testing the user’s actual surroundings, so you can effectively test their user experience.
• Improves Deployment Testing Accuracy – When it comes to testing new functionality, there is no better method than to do it in the same environment where it will be deployed. This is especially true when testing in lower-level contexts, which can have inaccurate data or different configurations. When switching between environments, might lead to irregular production deployments. When you test in production, you can be confident that your customers will experience the exact functionality that was tested.
• Testing and discovering how customers use a certain component – Production testing broadens the definition of testing. Instead of a “does this function or not” situation, testing might also imply experimenting and understanding how customers react to a certain component. We may utilize the testing in production technique to first test features as we did previously during sprint activities (Let’s call it phase 1). Following verification, we may do A/B testing and use metrics to collect data on real customers’ responses to the new product (This may address as phase 2).
• Increases the frequency of deployments (With Toggling) – To test in production, your organization will most likely need to adjust its overall perspective on how deployments operate. The days of decades between deployments with many features introduced in each deployment are long gone. These kinds of deployments are inherently risky, and frequently result in the “good enough” cause, with defects discovered early on that take months to fix in the following deployment.
  You’ll be more agile if you deploy more frequently. You may respond to customer requests more quickly, implementing updates as needed. Frequent deployments enable flag-driven development, which means that development will be done with the idea of having a feature flag (or a Toggle) that switches on the capability when applicable. If done effectively, this eliminates the need for a programmer to be concerned about functionality “leaking” into higher environments, even if the functionality isn’t ready.
• Production Testing as the Future – Many organizations are concerned about the notion of testing extending into production since testing has often certified the quality of the software before delivery to production. However, testing is evolving drastically as a result of agile and DevOps, and organizations must pick the best approach to adapt. The most promising approach is to involve the application at all stages of development and deployment.
• Beta Releases – Implementing production testing in your solution’s product beta version allows organizations to gain early feedback on newly released features. Beta versions also allow consumers to provide immediate feedback on concerns with the user experience.

Why Use Feature Flags (Toggling) in Production Testing?

Toggling, or feature flags, are crucial for efficient production testing. This is because they allow us to turn things on and off instantaneously against whatever condition we choose, without the complexity of a rollback or redeployment. Let’s look at what necessary capabilities feature flags provide in a production system that makes them so important:

• Enforce the stop of specific functionality: immediately turning things on and off without rollbacks or redeployments.
• User segmentation is the process of determining who has access to certain features, allowing you to perform specialized production tests or obtain specific production data.
• Progressive delivery entails testing with a limited set of production data (users, traffic, etc.) and gradually rolling out changes to bigger cohorts until the update is received by 100% of users in your production settings.

Here is how it is described in Wikipedia:

“A feature toggle in software development provides an alternative to maintaining multiple feature branches in source code. A condition within the code enables or disables a feature during runtime. In agile settings the toggle is used in production, to switch on the feature on demand, for some or all the users. Thus, feature toggles do make it easier to release often. Advanced rollout strategies such as canary rollout and A/B testing are easier to handle.
Even if new releases are not deployed to production continuously, continuous delivery is supported by feature toggles. The feature is integrated into the main branch before it is completed. The version is deployed into a test environment once, the toggle allows you to turn the feature on, and test it. Software integration cycles get shorter, and a version ready to go to production can be provided.
The third use of the technique is to allow developers to release a version of a product that has unfinished features. These unfinished features are hidden (toggled) so that they do not appear in the user interface. There is less effort to merge features into and out of the productive branch, and hence allows many small incremental versions of software.”

As seen above, feature flags decrease the cost of production testing to almost nothing. They make production testing easier and faster.

Why test in production rather than staging?

You may be asking why it is preferable to test in production rather than after staging. After all, the staging testing environment exists for a reason: to allow you to quickly test your products and guarantee that all features are working properly before going live. However, you should be aware that various people interpret the staging environment differently. Staging testing is an important process in most organizations and serves as a preliminary to the formal launch of the product. Unfortunately, faults in the staging environment are not always detectable.

Two main aspects contribute to this:

• The staging environment‘s size and complexity have a significant influence on overall testing effectiveness. Because the staging environment is frequently much smaller than the production cluster, configuration options for nearly every service will differ. This applies to the database and load balancer configurations, among other things. If these configurations are kept in a database or as a key value, auxiliary systems must be configured in the staging environment to ensure that it interacts with these resources in the same way as the production environment. Improper staging environment monitoring – Even if monitoring is sufficient, staging monitoring indicators may be erroneous since they must be monitored in an entirely different setting from the production environment.
• The staging environment is most likely to be insufficient. Even the most stringent monitoring measures cannot eliminate the possibility of erroneous data since the staging environment is almost certainly different from the production environment.

Even though the staging environment is as near to production as possible, some forms of testing are just best done with real traffic. Soak testing, for example, is a type of deployment testing that involves assessing or monitoring the reliability and stability of a service over time (For example CPU utilization, memory leaks, and so on). This is always done during the production process. This is when production testing is necessary.

What kind of production testing are available?

There are a few strategies that may be employed; let’s go through the most important ones.

Scale/Volume Testing

Although high user traffic may be mimicked in a lower setting, volume testing in a production environment yields the most accurate findings. It gives data for every type of network load, browser responsiveness, and server performance when serving client requests.

A/B Testing

A/B testing is one of the testing procedures that can only be used in a live setting. It involves releasing two (or more) different versions of an application/software to see if real users prefer one over the other. It is a statistical activity in which we divide the user base into two groups, A and B. Then, show group A users the base version and group B users a slightly modified version. Finally, we may compare both sets of user groups to forecast application behavior so that we can make an informed decision about which version to push out in the upcoming release. Because A/B testing can only be successful with real users, it cannot be performed outside of the production environment. When implemented right, it delivers somewhat input to the business.

Monitoring And Evaluation

Continuous monitoring is another set of test methods in production that can assist uncover vulnerabilities that can only emerge in production. Organizations may uncover software faults that can only emerge in production by continuously monitoring a prod environment following software release. Identifying sluggish loading pages in a web application, for example, changes drastically in production. This is because a website may load quickly in staging with a smaller data set and less traffic. It’s a whole different tale in production. Monitoring how long it takes a proxy server to execute requests provides real-world data on what visitors may encounter when using the site. Slow page loading is detrimental to the user experience and should be addressed as soon as feasible.

Regression testing

One of the most significant characteristics of a new release is that it does not hurt the current components. As a result, we must guarantee that no new bugs are introduced during the deployment and that the newly released code integrates properly with the pre-existing code. Regression testing in production may be a significantly reduced version of the entire regression test suite, depending on how the deployment is set up inside the organization.

To summarize, you should run as many regression tests in production as necessary to ensure that the deployment went well and that no unexpected bugs were discovered during the release.

Challenges in Testing in Production

While testing in production provides plenty of flexibility and opportunity, it is not an encouragement for testers to put off their testing until after the release. Quality has evolved to require additional team members, such as developers, designers, and architects, to test in all circumstances, including live or staging environments. The genuinely dedicated tester will use this to free up cycles and take on greater and better challenges.

It may also be enticing to adopt testing at the organizational level to promote a faster time to market at a lower cost, but this approach should be avoided at all costs. Product quality, user loyalty, brand acceptance in the marketplace, and the general status of the test team may weaken.

When properly implemented, testing in production should be considered extremely valuable, but risky when trespassed upon unprepared. Testing in production, with well-defined limits and a clear value proposition, has a lot to offer in the next years, especially as the lines between the product team and end users become increasingly blurred.

Let’s have a look at some of the challenges you may face to test in production.

• • Security Risks – When dealing with production testing, one of the most complex subjects to manage is security. We are no longer dealing with fake data, but with actual live data. This fact emphasizes the need of treating data correctly, which means we must exercise caution while testing with actual user data. Data protection laws are frequently stringent. Some software requires HIPAA compliance and imposes severe penalties for infractions. Others, particularly in the financial field, have a lot of personally identifying information (PII). Data leaking can lead to massive lawsuits and other serious consequences. Consider restricting the number of programs that have access to this data—the more locations this PII is kept, the greater the potential impacts for your company may be.
  • Cleaning up test data – non-development environments execute tests using simulated data. It is discouraged to utilize such data during production testing since it will compromise the integrity of production data and might be challenging to remove (Attempting to automate this removal using tools and scripts can be dangerous and result in the loss of vital production data.).
  • Required Deployment Readiness Skills – Testing in production necessitates the presence of a pretty advanced deployment mechanism. First and foremost, you must be able to deploy rapidly. You’ll have already moved away from largely manual deployments, which are risky due to instability. Following that, as previously said, you’ll need to perform more often deployments to properly integrate feature flags into your application. You must be able to use feature flags dynamically, which implies not just turning features on and off, but also modifying functionality dependent on the user. Finally, these identical flags must be present in case a feature flag has to be gently turned off. Managing feature flags at mass necessitates solid governance that works in tandem with your continuous integration/continuous deployment (CI/CD) pipelines, which is also required if you wish to test in production. It is vital to keep an enterprise-wide view of feature flag states.
  • Breaking production – A test in a production environment, like any other, will either pass or fail. A test that fails or does not proceed as planned might have unintended consequences in the production environment, such as breaking production and causing downtime.
  • The test window might be too small – A test that must run over a long length of time, such as load testing, will be interrupted, and its progress will be skewed by system maintenance tasks such as Archiving, defragmentation, and backups. These disruptions might result in misleading failures and inaccurate testing.

David Tzemach

The founder and owner of the Agile Quality Made Easy blog (25K followers). A platform that he uses for teaching and coaching others, sharing knowledge with people, and guiding them towards success while giving them the inspiration and tools to discover their own path.

See author's profile