And the Winner Is: Aggregate Model-based Testing
Istvan Forgacs
Posted On: October 27, 2022
9232 Views
8 Min Read
In my last blog, I investigated both the stateless and the stateful class of model-based testing. Both have some advantages and disadvantages. You can use them for different types of systems, depending on whether a stateful solution is required or a stateless one is enough. However, a better solution is to use an aggregate technique that is appropriate for each system. Currently, the only aggregate solution is action-state testing, introduced in the book Paradigm Shift in Software Testing. This method is implemented in Harmony.
Where can we use Action-state testing?
Action-state testing can be used for both stateful and stateless cases or even a mixed solution is possible if a long test case consists of inner states at some test steps, but at some other steps don’t.
The ‘action-state’ model is textual, containing ‘action-state’ steps. Test cases are generated from the action-state model. An action-state step is a triple containing
- a (user) action,
- a (system) response,
- the inner (test) state.
Actions and responses must be implementation-independent and as abstract as possible. Action is described at a high level so that humans can understand. For example, an action can be as:
add items so that price remains below 10
This is more abstract than adding two more concrete actions:
- add item for 7
- add item for 2
Even more, concrete actions would be:
- click on the ‘+’ icon next to Pizza Mexicana
- click on the ‘+’ icon next to Coke
However, these are implementation-dependent actions and can be created when the implementation is ready. Making an implementation-independent model is a huge advantage as it can be done before coding and the team can validate it against the specification detecting almost all the problems in it.
Modeling is easy: a subsequent step (step 2) is executed immediately after a step (step 1) if step 2 is indented to the right:
step 1
step 2
If two steps’ indentation is the same then they will be executed in different test cases:
step A
step B.
step A may contain an action and a response (stateless step):
action A => response A
or may contain a triple involving the inner state:
action A => response A STATE state for A
Models can be easily mapped with the requirements:
R1 my requirement A:
action A => response A STATE state for A
One model can embed other models. In this way, very complex systems can also be modeled.
You can easily build a model related to a feature or user story. From this model, the tool can generate the statechart and the test cases immediately. The statechart is another model representation, see later. By applying statecharts, you can easily detect missing actions and states. On the other hand, a statechart contains less information. The information on whether a transition is traversed more times is missing. For example, entering the wrong PIN for the first and second time is modeled in this way:
Add wrong PIN => wrong PIN added STATE waiting for PIN
Add wrong PIN => wrong PIN added STATE waiting for PIN
The related statecharts are the same independently of only once or twice the wrong PIN was added. Action-state models contain more information than statecharts and this is the reason why this technique doesn’t need guard conditions and consequently coding.
However, to avoid coding the models are a little bit larger.
You can also set an appropriate test selection criterion. A tool can offer the missing action-state steps to satisfy the criterion. There may be steps to be offered that are invalid. Fortunately, you can easily recognize an invalid step and it can be rejected. Accepting or rejecting an offered step is the replacement of adding guard conditions and some other coding. It’s much easier and non-coder testers can also do it.
Let’s consider the same requirements as in my previous blog:
The most difficult part is how to find appropriate states. A reasonable solution is that if two actions are the same, but the system calculates the response in different ways, then we arrived at different states. Here is an example. The initial state is ‘No discount, no bike’. From here we can add a car for which we go back to the initial state as no discount happens. Adding a second car, the calculation is the same as a car added to the cart. However, when ad the third car, the calculation will be different as a bike is also added for free. Thus we arrived at a new state ‘Discount, bike added ’. If we start with adding a bike and then adding three cars in a row, then there is another different calculation as the bike becomes free. Thus this is a new state: ‘Discount, bike converted ’. It’s obvious that different calculations can be tested as any may contain an error. In this way, you can select a minimum number of test states.
Adding a bike at the same level as the first car, we go to another state ‘No discount bike included’. All these steps cover requirement R1. This means that covering a single requirement with a single test is usually not enough. Here is the first version of the model:
|
1 2 3 4 5 |
INITIAL STATE No discount, no bike R1 The customer can add cars or bikes one by one: add car => one car STATE No discount, no bike add car => two cars STATE No discount, no bike add bike => bike added STATE No discount bike included |
and the related statechart:
You can see that the statechart doesn’t show that we added two cars.
For covering requirement R2, we should delete a car/bike that should be a child step of a parent step where at least one car/bike is available:
|
1 2 3 4 5 6 7 8 9 |
INITIAL STATE No discount, no bike R1 The customer can add cars or bikes one by one: add car => one car STATE No discount, no bike add car => two cars STATE No discount, no bike R2 The customer can remove cars or bikes one by one: delete car => one car STATE No discount, no bike add bike => one bike STATE No discount bike included R2 The customer can remove cars or bikes one by one: delete bike => cart empty STATE No discount, no bike |
To cover R3a, we should add two cars to the single bike:
|
1 2 3 |
add bike => one bike STATE No discount bike included R3a discount for EUR 700 when bike converted free: add two cars => two cars and a bike STATE Discount, bike converted |
R3b can be covered if we add a third car:
|
1 2 3 |
add car => two cars STATE No discount, no bike R3b For discount a bike is added when only cars are in the cart: add car => three cars and a bike STATE Discount, bike added |
Now it’s reasonable to continue with R4. To cover it we remove the third car:
|
1 2 3 |
add car => three cars and a bike STATE Discount, bike added R4 going below the discount limit it is withdrawn: delete car => two cars STATE No discount, no bike |
Considering R3c, we should add the deleted car again:
|
1 2 3 |
delete car => two cars STATE No discount, no bike R3c reaching the limit again, previous discount back: add car => three cars and a bike STATE Discount, bike added |
Our last requirement is R3d. We cover it by deleting the second car, adding a bike, and adding the second car again:
|
1 2 3 4 5 |
delete car => two cars STATE No discount, no bike R3d reaching the limit again adding a bike before: delete car => one car STATE No discount, no bike add bike => one car and a bike STATE No discount bike included add car => two cars and a bike STATE Discount, bike converted |
The whole model is the following:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
INITIAL STATE No discount, no bike R1 The customer can add cars or bikes one by one: (1) add car => one car STATE No discount, no bike add car => two cars STATE No discount, no bike (2) R3b For discount a bike is added when only cars are in the cart: add car => three cars and a bike STATE Discount, bike added (3) R4 going below the discount limit it is withdrawn: delete car => two cars STATE No discount, no bike (4) R3c reaching the limit again, previous discount back: add car => three cars and a bike STATE Discount, bike added (5) R3d reaching the limit again adding a bike before: delete car => one car STATE No discount, no bike add bike => one car and a bike STATE No discount bike included add car => two cars and a bike STATE Discount, bike converted (6) R2 The customer can remove cars or bikes one by one: delete car => one car STATE No discount, no bike add bike => one bike STATE No discount bike included (7) R3a discount for EUR 700 when bike converted free: add two cars => two cars and a bike STATE Discount, bike converted (6) R2 The customer can remove cars or bikes one by one: delete bike => cart empty STATE No discount, no bike |
Hurray, we covered the requirements with only five test cases (the ones (1) – (5)). Are we ready? Let’s see. The related statechart is here:
It’s a great help as missing actions can be detected easily. For example, from the state ‘Discount, bike added’ we can delete or add a bike. Considering ‘No discount bike included’, there is a missing edge going to itself, when deleting a bike from two. We can also add bike going to itself and going to ‘Discount, bike converted’. Finally, considering ‘Discount, bike converted’, there are several missing actions, i.e., adding/deleting a car and adding/deleting a bike to itself, delete bike going to ‘No discount no bike’ and delete car/bike going to ‘No discount bike included’.
Traversing each valid action/edge is the minimum test selection criterion, thus just covering the requirements is way insufficient. Involving the missing actions, we get the following improved model:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
INITIAL STATE No discount, no bike R1 The customer can add cars or bikes one by one: add car => one car STATE No discount, no bike add car => two cars STATE No discount, no bike R3b For discount a bike is added when only cars are in the cart: add car => three cars and a bike STATE Discount, bike added add bike => three cars and two bikes STATE Discount, bike added delete bike => three cars STATE No discount, no bike R4 going below the discount limit it is withdrawn: delete car => two cars STATE No discount, no bike R3c reaching the limit again, previous discount back: add car => three cars and a bike STATE Discount, bike added R3d reaching the limit again adding a bike before: delete car => one car STATE No discount, no bike add bike => one car and a bike STATE No discount bike included add car => two cars and a bike STATE Discount, bike converted R2 The customer can remove cars or bikes one by one: delete car => one car STATE No discount, no bike add bike => one bike STATE No discount bike included add bike => two bikes STATE No discount bike included add five bikes => seven bikes STATE Discount, bike converted add bike => eight bikes STATE Discount, bike converted delete bike => seven bikes STATE Discount, bike converted delete bike => six bikes STATE No discount bike included delete bike => one bike STATE No discount bike included R3a discount for EUR 700 when bike converted free: add two cars => two cars and a bike STATE Discount, bike converted delete car => one car and a bike STATE No discount bike included add car => three cars and a bike STATE Discount, bike converted delete car => two cars and a bike STATE Discount, bike converted delete bike => two cars STATE No discount, no bike add bike => two cars and two bikes STATE Discount, bike converted R2 The customer can remove cars or bikes one by one: delete bike => cart empty STATE No discount, no bike delete bike => cart empty STATE No discount, no bike |
The number of test cases increased from 5 to 13. The new statechart is the following:
Now all the actions are involved. We have seven requirements and we carefully covered by 13 test cases. You can think that 13 tests are too much, but it depends on the risk analysis. If the risk is high, you should test it carefully, otherwise tricky bugs remain undetected. You cannot merge cars and bikes as vehicles as they behave differently, only bikes are added freely or converted for free, and cars are not.
Considering the statechart, if you simply traverse it, some invalid tests are generated. For example, adding a bike from the initial state, then adding five bikes we arrive at ‘Discount, bike converted’. However, having only six bikes we have no discount, a contradiction. The state action-state model consists of only valid steps.
As mentioned, by applying a stronger test selection criterion, new steps are offered. For example, when applying all-transition-pairs criterion the following step is offered:
This is because the edge/action pair (add car, add bike), where add car is the action starts from and arrives at the initial state hasn’t been covered. If the risk is very high you should add these steps as well.
Conclusion
Action-state testing is an aggregate MBT method. This is the only one that requires no coding at all, just modeling. The model can be converted to a statechart, by which the original model can be improved and made complete. It can be widely used, and tricky bugs can be detected. From the models, abstract test cases are generated that can be manually executed. The model consists of implementation-independent steps, thus it’s a true shift left MBT method.
Author’s Profile
Istvan Forgacs
István Forgács PhD is an entrepreneur, a test expert and an author. He is the lead author of the book Practical Test Design and Paradigm Shift in Software Testing and the co-author of the Agile Testing Foundations. He is the creator and key contributor of the test-first, codeless test design automation tool Harmony. With his co-author Prof. Attila Kovács, they created a website that is a unique place where testers can exercise test design by executing and improving their tests. They introduced three test design techniques: combinative testing, action-state testing and general predicate testing.
Blogs: 8
Got Questions? Drop them on LambdaTest Community. Visit now
