And the Winner Is: Aggregate Model-based Testing

Istvan Forgacs

Posted On: October 27, 2022

view count7564 Views

Read time8 Min Read

In my last blog, I investigated both the stateless and the stateful class of model-based testing. Both have some advantages and disadvantages. You can use them for different types of systems, depending on whether a stateful solution is required or a stateless one is enough. However, a better solution is to use an aggregate technique that is appropriate for each system. Currently, the only aggregate solution is action-state testing, introduced in the book Paradigm Shift in Software Testing. This method is implemented in Harmony.

Where can we use Action-state testing?

Action-state testing can be used for both stateful and stateless cases or even a mixed solution is possible if a long test case consists of inner states at some test steps, but at some other steps don’t.

The ‘action-state’ model is textual, containing ‘action-state’ steps. Test cases are generated from the action-state model. An action-state step is a triple containing

  • a (user) action,
  • a (system) response,
  • the inner (test) state.

Actions and responses must be implementation-independent and as abstract as possible. Action is described at a high level so that humans can understand. For example, an action can be as:

add items so that price remains below 10
This is more abstract than adding two more concrete actions:

  1. add item for 7
  2. add item for 2

Even more, concrete actions would be:

  1. click on the ‘+’ icon next to Pizza Mexicana
  2. click on the ‘+’ icon next to Coke

However, these are implementation-dependent actions and can be created when the implementation is ready. Making an implementation-independent model is a huge advantage as it can be done before coding and the team can validate it against the specification detecting almost all the problems in it.

Modeling is easy: a subsequent step (step 2) is executed immediately after a step (step 1) if step 2 is indented to the right:

step 1
step 2

If two steps’ indentation is the same then they will be executed in different test cases:

step A
step B.

step A may contain an action and a response (stateless step):

action A => response A

or may contain a triple involving the inner state:

action A => response A STATE state for A

Models can be easily mapped with the requirements:

R1 my requirement A:

action A => response A STATE state for A

One model can embed other models. In this way, very complex systems can also be modeled.

You can easily build a model related to a feature or user story. From this model, the tool can generate the statechart and the test cases immediately. The statechart is another model representation, see later. By applying statecharts, you can easily detect missing actions and states. On the other hand, a statechart contains less information. The information on whether a transition is traversed more times is missing. For example, entering the wrong PIN for the first and second time is modeled in this way:

Add wrong PIN => wrong PIN added STATE waiting for PIN
Add wrong PIN => wrong PIN added STATE waiting for PIN

The related statecharts are the same independently of only once or twice the wrong PIN was added. Action-state models contain more information than statecharts and this is the reason why this technique doesn’t need guard conditions and consequently coding.

However, to avoid coding the models are a little bit larger.

You can also set an appropriate test selection criterion. A tool can offer the missing action-state steps to satisfy the criterion. There may be steps to be offered that are invalid. Fortunately, you can easily recognize an invalid step and it can be rejected. Accepting or rejecting an offered step is the replacement of adding guard conditions and some other coding. It’s much easier and non-coder testers can also do it.

Let’s consider the same requirements as in my previous blog:

how to find appropriate states

The most difficult part is how to find appropriate states. A reasonable solution is that if two actions are the same, but the system calculates the response in different ways, then we arrived at different states. Here is an example. The initial state is ‘No discount, no bike’. From here we can add a car for which we go back to the initial state as no discount happens. Adding a second car, the calculation is the same as a car added to the cart. However, when ad the third car, the calculation will be different as a bike is also added for free. Thus we arrived at a new state ‘Discount, bike added ’. If we start with adding a bike and then adding three cars in a row, then there is another different calculation as the bike becomes free. Thus this is a new state: ‘Discount, bike converted ’. It’s obvious that different calculations can be tested as any may contain an error. In this way, you can select a minimum number of test states.

Adding a bike at the same level as the first car, we go to another state ‘No discount bike included’. All these steps cover requirement R1. This means that covering a single requirement with a single test is usually not enough. Here is the first version of the model:

and the related statechart:

related statechart

You can see that the statechart doesn’t show that we added two cars.
For covering requirement R2, we should delete a car/bike that should be a child step of a parent step where at least one car/bike is available:

To cover R3a, we should add two cars to the single bike:

R3b can be covered if we add a third car:

Now it’s reasonable to continue with R4. To cover it we remove the third car:

Considering R3c, we should add the deleted car again:

Our last requirement is R3d. We cover it by deleting the second car, adding a bike, and adding the second car again:

The whole model is the following:

Hurray, we covered the requirements with only five test cases (the ones (1) – (5)). Are we ready? Let’s see. The related statechart is here:

related statechart

It’s a great help as missing actions can be detected easily. For example, from the state ‘Discount, bike added’ we can delete or add a bike. Considering ‘No discount bike included’, there is a missing edge going to itself, when deleting a bike from two. We can also add bike going to itself and going to ‘Discount, bike converted’. Finally, considering ‘Discount, bike converted’, there are several missing actions, i.e., adding/deleting a car and adding/deleting a bike to itself, delete bike going to ‘No discount no bike’ and delete car/bike going to ‘No discount bike included’.

Traversing each valid action/edge is the minimum test selection criterion, thus just covering the requirements is way insufficient. Involving the missing actions, we get the following improved model:

The number of test cases increased from 5 to 13. The new statechart is the following:

new statechart

Now all the actions are involved. We have seven requirements and we carefully covered by 13 test cases. You can think that 13 tests are too much, but it depends on the risk analysis. If the risk is high, you should test it carefully, otherwise tricky bugs remain undetected. You cannot merge cars and bikes as vehicles as they behave differently, only bikes are added freely or converted for free, and cars are not.

Considering the statechart, if you simply traverse it, some invalid tests are generated. For example, adding a bike from the initial state, then adding five bikes we arrive at ‘Discount, bike converted’. However, having only six bikes we have no discount, a contradiction. The state action-state model consists of only valid steps.

As mentioned, by applying a stronger test selection criterion, new steps are offered. For example, when applying all-transition-pairs criterion the following step is offered:

test selection criterion

This is because the edge/action pair (add car, add bike), where add car is the action starts from and arrives at the initial state hasn’t been covered. If the risk is very high you should add these steps as well.

Conclusion

Action-state testing is an aggregate MBT method. This is the only one that requires no coding at all, just modeling. The model can be converted to a statechart, by which the original model can be improved and made complete. It can be widely used, and tricky bugs can be detected. From the models, abstract test cases are generated that can be manually executed. The model consists of implementation-independent steps, thus it’s a true shift left MBT method.

Author Profile Author Profile Author Profile

Author’s Profile

Istvan Forgacs

István Forgács PhD is an entrepreneur, a test expert and an author. He is the lead author of the book Practical Test Design and Paradigm Shift in Software Testing and the co-author of the Agile Testing Foundations. He is the creator and key contributor of the test-first, codeless test design automation tool Harmony. With his co-author Prof. Attila Kovács, they created a website that is a unique place where testers can exercise test design by executing and improving their tests. They introduced three test design techniques: combinative testing, action-state testing and general predicate testing.

Blogs: 8



linkedintwitter

Test Your Web Or Mobile Apps On 3000+ Browsers

Signup for free