Calum McHaffie reviews James Bach’s presentation of Agile testing quadrants at the Canterbury Software Cluster event in Christchurch in July…
I must say that I’ve never really spent much time looking at either Marick’s or Crispin and Gregory’s version of the Agile testing quadrants before. So when James Bach presented his and Michael Bolton’s updated version, it made me think about how easy it is to just accept an idea.
The limitations of the old Agile testing quadrants
Supporting Programming vs. Supporting the Team vs. Critiquing the Product
James pointed out that he first heard the idea of the Agile testing quadrants when Brian Marick posed the idea to him. He found them mildly misleading, but thought it harmless. When Crispin and Gregory incorporated it into their Agile testing book, they modified it slightly.
This slight modification would appear to have a significant effect on separating the idea of agility and testing. The change was to the wording on the left side of the matrix – ‘Support Programming’ to ‘Supporting the Team’.
The differentiation of supporting programming and critiquing the product can be comfortably seen as developing and testing – naturally different tasks and make sense being on opposite sides of the matrix.
By adjusting the wording to ‘Supporting the Team’, Crispin and Gregory give the impression that critiquing the product (or testing) is not supporting the team. I (and James) find this idea absurd. Of course critiquing the product is supporting the team. The whole idea of Agile is to build a product, identify the weaknesses of the product (by critiquing it) then rebuild, critique, rebuild… and so on. How can critiquing not be supporting the team? The metaphor James used of comparing this to a football team is one I will never forget. Strikers (devs) have a role. They score goals (develop the product). But you would never field a team without a keeper (tester). Each position has its different role and responsibilities, but all are needed and all support the team.
Business vs. technology-facing
The other two sides of the matrix reference whether the effort is business or technology facing.
James posed that there is no need to differentiate between ‘facings’, based on the core heuristic of Agile – “Continually re-focus on value (in order to produce value) and ply our craft in ways that reduce the cost of change (rather than deny change)”.
More simply put, the business may not appear to care about how well the product is developed or how easy it is to test, but in the long run they must, as it makes it more efficient and cost effective.
The Agile testing quadrants ‘as it should have always been’
The new quadrants put more of a focus on the flow between the quadrants, as opposed to the defined stringent roles the old matrix had. Straightaway, I was drawn to the words in each corner – ‘experiment’ -> ‘design’ -> ‘build’ -> ‘test’ -> and back to ‘experiment’.
To me, it had already become more Agile just by making it more about flow than defining roles.
Each quadrant has its own set of tasks that inter-relate with the tasks from other quadrants. Then, on top of that, it also puts a set of tasks/goals in the middle that the whole team strives for.
For me, this pulls the quadrants together nicely, showing that although there are different tasks suited to different roles, they are all intertwined and work together to focus on a common goal.
Overlay 1 – Analysing/Synthesising/Value of Product/Cost of Development
Throughout James’ presentation, he illustrated ideas over the quadrants that he believes can be seen through the tasks assigned in the quadrants. From the first overlay he presents, you can see there are still elements of the original quadrants in their design, such as the ideas of analysing vs. synthesising and high value of product vs. low cost of development – both of which you could say is a return back to Marick’s original design (analysing=critiquing, synthesising=supporting programming, high value of product=business facing and low cost of development=technology facing).
Overlay 2 – Focusing/Defocusing/Envisioning Success/Anticipating Failure
The second overlay shows the mentality the team uses when in each quadrant – envisioning success when you are designing, focusing when you are building, anticipating failure when you’re testing and defocusing when you’re experimenting.
While I do find myself thinking some of these mentalities would be suited to multiple areas, I have to agree with them in principle. And when you take these mentalities and compare them to the wording along the border of the matrix, I find myself thinking it just makes even more sense.
“Discover something worth building. As we do so, we develop the design (envisioning success), so that we can build some of it. As we do so, we build cleanly and simply (focusing) so that we can build with change in mind. As we do so, we foster testability (anticipating failure) so that we can study what we built. As we do so, we experiment imaginatively and suspiciously (defocusing).”
Overlay 3 – Central Obstacle Divides Work
The third overlay was their answer to “Can a developer test their own code?”, their answer being “Yes… but it takes time”. There is a difference in the mindset required to develop a solution and then to test it (to any decent extent).
My knee-jerk reaction (probably the dev in me coming out) was to argue against this, but the more I thought about it, the more I saw the relevance.
The first question you may have is, “What is the difference in mindset?” Here are my thoughts:
- Developer – How can I make this work?
- Tester – How can I break this?
The second then may be, “Why would it take time to switch mindset?”
This is the point where I start to have an internal argument. The dev in me saying “Pshh, easy!”, the tester saying “Critical distance, you’re too close”.
Thinking back on projects I have worked on in the past, I can recall examples for both cases. But the one thing that may define the difference for me is complexity. The more complex the product, the harder it is to switch mindset.
I recall a project I worked on at university where we had to write our own messaging protocol. We got it working in no time at all and were quite proud of what we had done. We had even written unit tests and a few basic process tests. Then our nominated tester came across and proposed a few more tests to run, which of course we had overlooked and had to revisit.
Of course, had we spent more time looking, I’m sure we would have identified these tests and eventually found the bugs. But it was notable that having someone else there to test gave us a much faster turnaround on bug identification and resolution.
Note: By testing here, my interpretation is that we are talking more about UAT and process testing, rather than unit testing (which I would consider a developer task).
Overlay 4 – Critical Distance
The fourth overlay is about critical distance. I thought this one was probably less about the quadrants, but still a great way of showing the need for testers.
The first question a product owner may have is, “Why not get the developers to test the product?” This has already been covered above by Mt. Mindset.
The second question a product owner may have is, “Why don’t we just let the public test the product?”. James’ answer to this is, “Ask the developer/business owner to try it. Then watch them try to understand the incoherent abusive results they get from disgruntled customers”. A tester is good at identifying bugs and conveying the bug to the business/developer in a clear and concise fashion.
Other noteworthy thoughts
Testing vs. checking
Although not specifically to do with the Agile testing quadrants, James made a point that I believe is worth mentioning. The question is often asked “Why don’t we get rid of all our testers and automate all our testing?”. You never hear someone say “Why don’t we get rid of all our developers and automate all the development?”.
The reason James posed is that developers do the smart thing and call their automated work ‘compiling’. Why do we not, as testers, differentiate between testing that requires thought (testing that can’t be automated) and the checking that we do with automation?
While I do not completely agree with this, I do see the point that we have to make. There is some testing (or checking) that we can automate easily, but there are other tests that require more thought, that you wouldn’t or couldn’t automate. Automation will never replace exploratory testing.
You do not have to know how to code to test
During his presentation, James often got sidetracked on areas he seemed passionate about. Here is one worth mentioning.
His argument for not needing to be able to code to test is based on the following…
In Agile, there is this idea that everyone should be cross-functional. This defeats the whole purpose of being a specialist. Why would you study and strive to be good/great/the best at something and then spend your time trying to do something else?
Given this logic, a tester would not need to be able to code to test, they need to know how to test.
That said, I see multiple arguments that say it is very beneficial to know how to code or at least be able to read code.
- If you understand the way a developer thinks or are able to read their code, it bridges the communication gap that can sometimes exist
- Taking the Agilist view, if you are cross-functional and able to code, you have the ability to assist the developers when the demand is there
Coming away from the presentation, I found myself thinking that none of the ideas proposed were particularly new, but the way they visualised it and the way James presented it was a clean way of pulling it all together.
Overall, I thought the new testing quadrants James and Michael proposed were a vast improvement on the old models. (It would be interesting to hear a review from a developer or BA regarding these quadrants).
As for his actual presentation, the passion he has for the industry and the examples he used made it thoroughly entertaining and really made the message clear.
I would absolutely recommend you attend one of his presentations or chat to him if you have the opportunity.
There is much more to these new quadrants and James’ presentation than I have mentioned here, so if you’d like to discuss it over a beer or two or three, let’s meet up.
James Bach’s slide presentation on ‘The REAL Agile Testing Quadrants’ can be found here.