7 Ways Monitoring Data Can Aid in Testing Improvement

We are living in the era of microservices and big data, particularly in the context of web services. Monitoring and testing in production are critical strategies for scaling up modern software systems. But how does testing monitoring fit into the bigger picture? What should testers do to use monitoring, and how can it assist us? The discovery of monitoring systems had a significant influence on my work as a Quality Director. I discovered a great deal about the product, its consumers, and even my own procedures. It greatly aided me in managing my prejudices and greatly improved my testing work. Assume you have a website that receives eight million visitors day after day. How can you ensure that it works not only on your brand-new work MacBook but also for a customer running Windows 11 and using an outdated browser? The ability to monitor might be beneficial to you.

Here are seven key advantages of monitored data for testers that you can use to motivate your team to use monitoring—as well as experience for yourself.

Determine the value of bugs

Do you comprehend the implications of the bug you just discovered? I can answer this question since I have a monitoring setup, which offers me additional room to gather useful information. Members of my group began referring me to specific issues identified by others to enquire about their impact, rather than simply shaking their heads since yet another problem had been uncovered. With a complete monitoring system in place, I can duplicate the situations utilizing Databases on observed data, letting me establish how many customers are affected in production.

Of course, this only applies to things that are currently in production, but it may be really useful. It was helpful for me to understand that some of the defects I believed were important only affected 10% of the customers. The ability to measure the effect of defects greatly simplifies development and prioritizing.

Learn about your customers

I can obtain a significant amount of information on the users by using monitoring and analytics. For instance, what browsers and gadgets do your product’s customers use? I was recently in an engagement where, using analytics, we discovered that the most used browser for the product was Opera. I was astonished that this was still true in 2022, especially because our team was planning to terminate support for this browser entirely. Monitoring enabled us to understand that our consumers are not always like us: they embrace technology in various ways.

Seeing statistics such as browser and device activity data, or even how users interact with our products, may be eye-opening. By seeing how people engage with the product, I can determine what the most often utilized functionality of the product is—and it is not necessarily what I anticipate. Monitoring allows me to better measure and understand the product and what is essential to consumers.

Highlight KPIs

Sometimes there is frustration because business and development teams do not speak the same language. However, monitoring assists me in bridging the gap between these sectors. First, I identify the team’s key performance indicators (KPIs). Different departments have different perspectives on what is essential. Sometimes I have to question KPIs since certain parts may not accurately reflect product performance, and we shouldn’t simply follow the figures. However, after the KPIs have been specified, I may work on displaying the KPIs by developing dashboards utilizing monitored data with particular time frames using tools. For example, because feature A was the most successful, I built a dashboard to measure user actions with it, such as the number of users that interacted with it over the weekend. Dashboards regarding the essential feature improved the team’s communication and helped to build a uniform language.

Use patterns to investigate problems

What I value most about monitoring is the capability it gives us to understand the problems that emerge. Using basic querying languages like SQL, I can track back the path of a specific issue, allowing us to uncover trends and locate more clients experiencing the same difficulties.

For example, I once discovered that I couldn’t use a key feature of our product in a specific workflow. I couldn’t recall the specific steps I followed, but monitoring assisted me in tracing my actions. After that, I could investigate the circumstance that produced the problem. I searched the system to find people who have conducted the identical scenario after making the situation as simple as possible. What I discovered was that not every user who followed the same procedures had the same result—for one group of users, the feature worked. When I looked at the data more closely and verified other user attributes, I discovered that the problem was tied to the customers’ location. Only a few locations were affected.

It may be both fascinating and illuminating to discover more about why problems have arisen. And the insights gleaned from monitoring data can greatly assist your development team in resolving issues.

Prioritize your testing

Understanding what features are truly important—based on facts, not what someone on the team told me or what I thought they were—enabled me to focus my tests effectively. I constructed queries utilizing observed data to assist me in answering the following sample questions:

• What features are the most popular among customers?
• Which browsers are the most popular among these customers?
• How do users interact with functionalities?

Once I’ve answered these questions, I’ve been able to move forward. For example, if mobile Firefox covers 75% of consumers, I must conduct significant testing beforehand. The most often used functionality, browsers, and devices should also be tested the most so that we can be confident in the quality of our product—and client happiness.

Examine regressions

When the customer base is extensive, your tests may not cover all possible regression cases. To make this situation less severe and frightening, you may consider dogfooding (internal testing of your product), and user testing (releasing only to selected customers who are willing to deal with defects). When a major update comes, especially if it is only for a subset of users, I track it and analyze user interactions in real-time monitoring dashboards that I developed. The fact that users will cover the combinations of devices and browsers and test the product in the manner they would use it is amazing.

During many releases, I normally run a quick manual exploratory test in production myself, and then I examine the real-time dashboards that show query results to track user interactions linked to the update that is being launched. I frequently filter the dashboards by environment or browsers. For example, even if I do not possess a device with Microsoft Edge mobile browser, I may test if Edge mobile browser users can do the same process and actions as users of other browsers without experiencing any functional gaps. I also often monitor certain periods if monitoring offers real-time data.

Configure alerts

I am not restricted to just monitoring the releases. I may also set up automated notifications that are generated when a certain threshold value is exceeded. Once I’ve confirmed and agreed on the KPI definitions, I can quickly set up alerts for those conditions—for example, that logged-in users should account for 30% of all users browsing a certain website on average every minute. Another example would be the number of people who use a critical functionality.

Real-time monitoring is beneficial for critical functions that contain business logic. I have some alerts set up that also send me an email so that I may research why the behavior occurred as soon as possible. Then, if it is confirmed and accepted as a real-time issue, we may acknowledge and resolve it as quickly as feasible.

When testing web servicesz, think about the fact that they may be available 24 hours a day, and traffic may interfere with alerting, so be sure the problem is present. I occasionally discover anomalies, but this raises the question of whether a KPI is reliable enough if it is unpredictable.

Finalizing

Many firms have no monitoring procedures at all, and those that do frequently lack specific components that you as a tester would find beneficial. You must investigate your product to see whether API queries, parameters, and so on might be logged. Then, convey the significance of these elements in a single universal location where you may see monitored data. It might be one of the monitoring tools with a web interface, or it could just be a database that you can query if you have your monitoring systems set up. Do not be scared to go above and above to have something checked. The facts will most likely be eye-opening—not just for you, but for the entire group.

David Tzemach

The founder and owner of the Agile Quality Made Easy blog (25K followers). A platform that he uses for teaching and coaching others, sharing knowledge with people, and guiding them towards success while giving them the inspiration and tools to discover their own path.

See author's profile