Maximizing Feedback for Developers with Continuous Testing

Developers need feedback on their work so that they know whether their code is helping the business. They should have “multiple feedback loops to ensure that high-quality software gets delivered to users”[1]. Development teams also need to review their feedback loops so that they can maximize useful feedback.

Testing provides feedback and should be continuous so that it provides quick and continuous feedback to the developers. Testing must always be happening as part of the development process. There are many different types of testing.

Continuous Integration as a Feedback Loop

An important feedback loop for the developers comes from continuous integration (CI). CI gives developers feedback on whether they have “built the thing right”. It provides an excellent way to give feedback to developers because each code change triggers a build process that includes running automated tests. “If any part of this process fails, developers fix it immediately.”[2] The tests in CI will give feedback that the developer’s code change has not broken existing functionality.

The automated tests in CI should include unit tests, which will check that each component does what it should. CI should also include linting, which will check the code for errors without running it. Integration tests, which check that components work together correctly, should also be part of CI. Automated tests that run through the application’s user interface (UI), and application programmer interface (API) will also give developers useful feedback and should be included in CI.

CI is also a useful feedback loop because it gives developers feedback on their code quickly. CI will give the developers quick feedback because the tests in CI run automatically when there is a code change. If the feedback comes to the developer later then they may be already working on something else and the feedback won’t be so relevant. If the feedback means that the code needs to be changed then the developer will need to stop what they are doing, reacquaint themselves with the code that needs to be changed, and then make the change. Changing the code would be easier and more efficient if the feedback had been quicker because the developer would have still been working on the code that needed to be changed. The additional work that results from late feedback is waste. Quick feedback helps to eliminate this kind of waste. Quick feedback also helps to motivate developers as they feel their work is valued.

The development team will also need a feedback loop on the performance of the application. If the performance of the product degrades, the developers will want to know and build into their work fixes for the degradation of performance. Performance tests should be automated, be part of continuous testing, and give feedback to the developers so that they are aware of changes in the performance of the application.

Developers should shift left on security testing [3] so that the team has a feedback loop for security. Automated security tests should be included in CI because this will give quick feedback to the developers. Continuous testing should include security testing so that developers are confident that their changes do not compromise the application’s security.

Ensuring Quality Feedback from Automated Tests

Developers need “excellent, rapid feedback”[4]. Excellent feedback means that the feedback from automated tests also needs to be of high quality. If there are automated tests that are flaky, and sometimes pass and sometimes fail, the feedback from the tests will be inconsistent. Flaky tests will undermine the confidence that developers have in automated tests because the developers will not know how to interpret the results of the tests. To ensure that the tests give quality feedback there needs to be investment in the maintenance of the automated tests. Flaky tests need to be removed from running quickly or be fixed.

An investment in the quality of the feedback from the automated tests will also help to develop a positive feeling in the development team because the automated tests will give developers the confidence to release their code. An Investment in maintaining automated tests is also an investment in people.

Accessibility testing should be included in continuous testing so that developers get quick feedback on how their code changes affect accessibility. There can be legal and commercial consequences if an application does not meet appropriate accessibility standards. Multiple test automation tools can be used to test that accessibility standards are maintained, and these automated tests should be included in CI. Manual testing is also required because automated tests for accessibility can miss accessibility issues. [5]

Code reviews provide a feedback loop to developers and should be regarded as part of the system of continuous testing. The development process should include code reviews of all code changes. Code reviews give feedback too as the code reviews give feedback from his/her peers to the developer who wrote the code.

Feedback for Quality

Exploratory testing also provides useful feedback and should be part of continuous testing. Testers will not need to execute manual regression tests because the regression tests have been automated and run as part of CI. This gives testers the space and time to perform exploratory testing on the latest builds from CI[1]. Exploratory testing will enable testers to use their imagination and knowledge of testing to provide helpful feedback to developers on the latest builds. Testing is too important to wait. To save the time of maintaining a test environment, the development team should have made architectural decisions that mean that most exploratory testing can be done without an integrated test environment. [7]

A feedback loop from production is also useful. This loop will provide information on issues that are occurring or have occurred in production and will enable developers to react to these issues.

Recently there has been a discussion in the DORA Community of Practice about the risk of focusing too much on output rather than outcome, and this applies to feedback too. The feedback from automated testing can be said to be about output because the results of automated tests check whether ‘the thing has been built right’. Feedback on the outcome of the code change would give feedback on whether ‘the right thing has been built’. Feedback on the outcome would view the code change from the perspective of the customer. Developers need feedback loops from continuous testing both on output and outcome.

An organization should work jointly with its customers[9], and so continuous testing should provide feedback from customers too. Customers are part of the system and the development team will benefit from having feedback loops from testing that enable them to adjust their work in response to feedback from customers. These feedback loops will tell them if they have built ‘the right thing’ and if the outcome of their code change will be positive.

Product Owners guide the teams that they work with and often organize user testing. Developers make decisions all the time which affect what functionality is developed and will make better decisions if they are aware of the results of user testing. Feedback will help a development team to develop what the customer wants more quickly. Developers can write code that passes all the automated tests and so have “built the thing right” but it may not be exactly what the customer wants or needs. Feedback loops from customers need to be part of the development team’s process.

A way of creating these feedback loops is to involve developers in the user testing process. A company I worked in a few years ago as Test Lead explicitly encouraged developers to get involved in user testing. This practice was done to help the developers understand customers so that when they were writing cards and writing code, they could consider customers’ needs. An additional feedback exercise that the company organized was to have a user testing week and to encourage all developers to attend at least one session of user testing and discuss what the session found. The understanding the company’s engineers had of the company’s customers, following their involvement in the sessions with the company’s customers, contributed to the company making a “successful exit”.

In addition to continuous testing, developers should be getting feedback from customers, sales and customer support. “Ongoing monitoring of both customer and field feedback and post-release defect data can provide valuable information”[10] for the future of the application. This insightful information will help developers in making decisions that positively affect customers.

Feedback loops from customers can also help the team feel good about their work because positive feedback from customers will boost the morale of the development team.Developers need “rapid feedback”[4]. If feedback is not rapid the benefits of reacting to it will be reduced, for example, customers may have started to churn before the developers can react.

CI provides quick feedback. To get even quicker feedback, developers should also be able to run the CI tests on their workstations to get feedback on the code they are working on.[7] If a developer can run the tests locally they can get feedback before they commit their code.

Implementing Deming’s Philosophy for Continual Improvement

The feedback loops needed from continuous testing will change over time as some feedback loops may cease to provide useful feedback, and new feedback loops may be required. To maximize feedback from continuous testing, the development teams need a method to review the feedback that they are getting.

The feedback loops that they have can be reviewed by using W. Edwards Deming’s System of Profound Knowledge (SoPK)[8]. The SoPK enables people to create a theory of what needs to be done by looking at psychology, statistical variation, and systems thinking. Developers do not have to be experts in these areas to use the SoPK. I recently had a conversation with Dennis Sergent, and he said that people should start where it has meaning for them.

When considering psychology to review the need for a feedback loop, the developers can consider if a feedback loop will help an individual developer or the development team to interact with one another. Considering statistical variation will help the developers understand how a feedback loop will affect measures of variation, such as deployment frequency. Systems thinking will help the developers view their development team as a system and consider how a feedback loop will help their development team as a system.

After considering psychology, statistical variation, and systems thinking, the developers should develop a theory of which new feedback loop is needed. This theory can then be implemented using the Deming Cycle[3]. This cycle has four parts: Plan, Do, Study, and Act. The team should start by creating a ‘plan’ on how to implement the feedback loop. They can then ‘do’ by implementing the new feedback loop, followed by a ‘study’ of the result of their action in creating the feedback loop. They should then ‘act’ on what they have learned from the ‘study’ because as a result of the study they may want to change the feedback loop, keep it, or remove it. The team will learn with each cycle of the Deming Cycle.

Conclusion

To maximize the feedback developers receive they need to have multiple feedback loops from continuous testing and to use Deming’s philosophy to review their feedback loops so that developers get the most useful feedback on their output. The feedback loops should help the developers to know if ‘the thing has been built right ‘ and help them understand the outcome of their code change so that they know if ‘‘the right thing has been built”.

References
[1] Accelerate by Nicole Forsgren PhD, Jez Humble and Gene Kim (2018, p73)
[2] Accelerate by Nicole Forsgren PhD, Jez Humble and Gene Kim (2018, p72)
[3] Accelerate by Nicole Forsgren PhD, Jez Humble and Gene Kim (2018, p227)
[4] Implementing Lean Software Development by Mary and Tom Poppendieck (2007, p177)
[5] Accessibility evaluation tools you should know about! By Peter Johnson (video)
[6] Accelerate by Nicole Forsgren PhD, Jez Humble and Gene Kim (2018, p90)
[7] Accelerate by Nicole Forsgren PhD, Jez Humble and Gene Kim (2018, p73)
[8] The New Economics by W. Edwards Deming (1994, p92)

Further learning:
eLearning: DemingNext from the Deming Institute