Part 2 – Test Consultant Amy Langbridge introduces Specification by Example to Static Testing
All too often, defects raised during test design and execution highlight gaps in the specification. During retrospectives, we notice that feedback initially provided during the static review process missed important scenarios altogether. Identifying solution gaps during testing highlights the inefficiency of the techniques used for static testing.
In the interest of bridging this gap, the static review phase conducted on a traditional project is an area where lean techniques could improve the effectiveness of the reviews conducted on the specifications prior to test design and execution.
Here, I aim to share an approach that was successfully applied during the review phase on a traditional project. I do not intend to re-invent the wheel, but rather to describe the context and the subsequent approach used that helped to identify solution gaps in specifications. In Part 1 last week, I described the context in which the approach was used and where it is placed in the software development lifecycle. In Part 2, I illustrate the lessons discovered through trial and error provided through a practical example.
The tester writes out scenarios using ‘given, when, then’ in an Excel spreadsheet.
a. ‘Given’ refers to the known state of the data required to do the test (ie. a description of the state in which the data needs to be in prior to the action being taken).
b. ‘When’ refers to the action that is required (i.e. not a sequence of test steps).
c. ’Then’ refers to the ultimate end result (within the context of the test) after the action is taken.
The scenarios intend to piece the requirements together ie. a scenario is not developed based on one requirement – one would be rewriting the specification. Instead, the aim is to bring the solution together as a whole where the document has separated it out. The specification provides the detail, the scenario provides a way to test the end-to-end solution without the code, document it, communicate it easily and is an aid to find solution gaps.
The scenarios use commonsense. That is, they are at the level of detail required to prove your point. They can be as simple and high level as shown in Table 1 or more detailed as shown in Table 2.
Ideally, the scenarios achieve the ‘end-to-end’ result within the context of the scenario. They are best communicated when they are written in the order in which they will occur as shown in Tables 1 and 2. Here, the row 1 ‘Then’ column leads onto (or informs) row 2 ‘Given’ column.
Where a ‘vanilla case’ scenario (ie. happy day scenario) is written as the first scenario, it acts as a starting point for the team. Once this is validated in the review session, it forms the foundation from which more complicated scenarios can follow. Discussing vanilla scenarios is important for the team to ensure everyone has the same understanding of the rules to start with before diving into scenarios that challenge the basic rules.
An extra column was added called ‘Team Outcome’ shown above which represents the decision made by the team during discussions as to whether the scenario was agreed as valid or whether any questions or follow-ups are required.
In order to reduce the amount of redundant detail communicated in each scenario, a list of assumptions could be provided at the start. For example, in Table 1, a list of assumptions that could be communicated are:
One way of identifying a scenario and its gaps would be to work backwards. That is, by answering the questions:
a) What is the end result that needs to be achieved?
b) What are the multiple different end results that can be achieved?
c) What scenario is required to achieve that end result?
d) What boundaries would expose different end results?
e) What end results are discussed in the document but need current processes (which are not documented) to produce them? (Answering these questions will help educate all team members involved who are not experts in the systems/processes involved)
f) What possible end results are not covered by the specification?
The scenarios are only written where the tester needs to prove a point. That is, the scenarios are not a repeat of the specification. For example, avoid writing low-level detailed scenarios showing the exact error messages that are expected. All one would be doing is repeating the information that is already in the specification. Instead, the scenario could state that an error message is displayed if one would deem it necessary to even write that scenario out and discuss it. In addition, the scenario could be written to prove one’s point that a specific requirement is untestable.
Through the review sessions, feedback is provided on the specification in the form of the scenarios. It is important to keep in mind though that feedback can either slow a project down or keep it moving forward. It can frustrate a team or keep them motivated. No document is perfect – any author of a paper already knows this. Every author who has had to piece together a complicated argument will find that certain feedback is fantastic to get. Some help to identify gaps in your arguments, while others leave you feeling like the reader has missed the core concept completely. This adaptation of SBE is about identifying the ‘valuable feedback’ ie. identifying the solution gaps. Identifying valuable feedback from the unnecessary feedback is also about picking your battles. Keeping the focus on the end-to-end solution within its context of the wider purpose of the solution helps with this. Finding that line of ‘good enough’ is tricky and something that can only be worked on as you move forward by doing your retrospectives (as a team and individually). That is, a good place to start is the question, “Should I have pushed more to have a certain thing fixed during static testing? Was a fuss made on an area that, in hindsight, was not really necessary and it actually pulled focus away from another area where a fuss should have been made?”.
It is important for the tester to ‘do their research’ on the business process as a whole before attempting to write or during the writing of scenarios. That is, source as much information as one can about the process. Source expert knowledge on the process, read as much as you can on the subject and, if you can, literally play around with the existing system to understand the existing business process and the changes to be implemented. Understanding the current system helps identify questions about the proposed new solution.
Grouping of scenarios
The scenarios are more easily communicated when they are grouped. That is, the nature of the business process guides the grouping of the scenarios. This feeds into test design. For example, in the case above, grouping that could be used are ‘integration scenarios’, ‘transition scenarios’.
Integration scenarios: These include both the system integration and system tests that need to be discussed. For example, the final end result may be that the customer receives an email confirming registration. However, it is also important to test a different condition that should result in the creation of email confirmation but it is not necessary to ensure the email is correct.
Transition scenarios: These could be developed due to the need to test across date boundaries – ie. scenarios where different outcomes are expected depending on the date. These could be important to discuss and ensure the team’s understanding of these are uniform.
The logical grouping of your scenarios aids your test design. The majority of your effort has already been put into, not only thinking about finding solution gaps, but also the best way to communicate your scenarios easily without having to jump around on different areas of the solution. Why not use this to inform your test design?
The grouping of the scenarios developed can inform the tickets used for the visual management/Kanban board, as well as the structure used in Quality Center. That is, the structure in QC, the visual management/Kanban board and the grouping of your scenarios all match one another. This aids communication to the wider project.
Once the list of scenarios is written, test would chair the review involving all members of the team responsible for delivering the product. These sessions are not ‘one-off’. Instead, they are run regularly as the solution changes. Test needs to play the role of mediator as well and encourage the whole team to brainstorm scenarios, while also keeping in mind the need to increase coverage, not redundant test cases.
The scenarios can then be used as a shell for your test cases. More test cases can be developed, but the test scenarios have been discussed and agreed as correct and can therefore be used in your test design and execution.
One way to determine the usefulness of the approach used for static testing is to compare the static defects found versus defects found in test execution. If the types of defects found during test execution (or heaven forbid, production defects!) are based on solution gap defects, then the approach used to identify and communicate these during static testing was not effective. The types of defects found in production or test execution can be used to inform the effectiveness of your initial analysis and therefore provide a starting point on the changes necessary to make the approach more effective.
Writing scenarios in the ‘given, when, then’ way does require the tester to make a judgement call as to the required level of detail and the type of scenarios written. Here individual retrospectives can help refine the technique used – that is, answering these questions after each review session could be useful:
Have I gone into too much or too little detail? Did I communicate my point well enough here?
Am I focusing on the end-to-end solution or am I just rewriting the specification?
Are these scenarios helping us find any solution gaps? What needs to change to make this effective?
Writing SBE scenarios in this way is not about rewriting the specification. Instead, it’s about bringing the solution together as a whole, while specifications – in order to accommodate the detail – pull the solution apart. It’s about highlighting aspects that are necessary to discuss as a team to ensure all have the same understanding of the solution. This approach is about doing effective analysis and increasing communication and collaboration between all team members earlier in the review phase on a traditional project.
 Prove your point: The scenario is only written if it is worthy of discussion as a team and the detail associated to it because the aim is not re-write the Specification.