On a recent assignment, I was responsible for testing changes in a System Integration Test (SIT) environment. All the JIRA tickets under test needed to remain in the ‘SIT Testing’ status column of the Scrum board for a couple of sprints. Only when the full set of JIRAs was considered ready, could the JIRA be moved into the next status column (i.e. UAT Testing). This meant that the SIT Testing column held JIRAs that needed to be tested, were being tested or had been tested.
The columns on the JIRA Scrum board looked something like this (text blurred for privacy reasons)…
I needed to find a way to provide visibility of the testing progress. For each SIT Testing phase, there were likely to be up to 100 JIRAs listed in the SIT Testing column – so a single list wasn’t much help.
A further complication was that the burn down for the development team focused on the first five columns (from ‘To Do’ to ‘Ready to Demo’) and the dev team needed to have full ownership of the Estimated Hours and the Remaining Hours fields available on the JIRAs. As a consequence, I couldn’t record Remaining Hours on my SIT Testing JIRAs in the normal way.
The problem: I needed to find a way to estimate my testing tasks and monitor progress and provide visibility for others.
The approach that I have been trialling – and that’s working quite well – is to have a dashboard that displays filtered lists of the SIT JIRAs based on label values that I’ve added onto them.
I’ve also been able to display the outstanding estimated hours by adding this value into an unused numeric field on each of the JIRAs and summing this value on the dashboard. (I recommend you ask for a ‘Remaining Hours to Test’ field to be added to your JIRA issue layout, but we’re using the unused Story Points field).
Here’s a compressed version of my new dashboard…
The labels used on the JIRAs
I came up with a list of labels that I thought would be appropriate to identify the status of each JIRA in the SIT Testing queue. These are…
Labels for the basic flow of testing
sit_prepare: This label is added to a JIRA when it first comes into the SIT Testing queue. It’s a reminder that I need to review the Acceptance Criteria and other details and estimate how long the testing will take.
sit_ready: A JIRA will have this label once the estimated hours have been added. The JIRA is now ready to be picked up by a tester.
sit_in_progress: This label is used once someone has begun work on the JIRA.
sit_passed: Success! Once a JIRA has passed testing, we formalise that by using this label.
sit_failed: When we find failures, we normally raise a new JIRA for the development team. But we want to be reminded that we will want to come back and do a sanity test of the original JIRA, once the finding has been fixed.
sit_with_others: Sometimes we find ourselves waiting for someone to review our queries or findings (or fix them) or to give us other information about the JIRA. We use this label to flag that we’ll need to ensure that we get this JIRA back to complete our SIT testing.
sit_on_hold: Occasionally we need to delay testing particular JIRAs and this label allows us to separate them out.
We also use a different set of labels to identify functionality area. This allows us to provide a heat map on the dashboard and helps us find JIRAs that may be good to test together or in sequence.
The SIT filters
We have a filter set up for each SIT testing status label, as we use the filters to provide our lists on the dashboard, plus a few variations.
Our ‘SIT – Ready to Test’ filter is a simple filter that will return all JIRAs for our project that have a status of ‘SIT Testing’ and a label of ‘sit_ready’.
The ‘SIT – Prepare test conditions/Initial read thru/Size’ filter is a bit more complicated as it needs to pick up anything that will need to be tested in SIT, but doesn’t have an appropriate sit label as yet. The filter we use is…
project = XXXX AND component = “Xxxxx" AND status = "SIT Testing"
AND (labels = EMPTY OR labels = sit_prepare OR labels not in (sit_ready, sit_on_hold, sit_in_progress, sit_passed, sit_failed, sit_with_others)) ORDER BY priority DESC”
The ‘SIT – All’ filter needs to pick up JIRAs that are in the SIT Testing queue and also those bug JIRAs that have been raised and will be deployed to SIT shortly. The filter used is…
project = XXXX AND component = “Xxxxx” AND (labels in (sit_in_progress, sit_passed, sit_ready, sit_with_others, sit_prepare, sit_failed, sit_on_hold) OR status = "SIT Testing") ORDER BY priority DESC
The gadgets on our testing dashboard
The most frequently used gadget on the dashboard is the Atlassian ‘Filter Results’. For each use, select the filter to be used to return the JIRAs, define the number of results to be displayed on the first ‘page’ of results and specify the JIRA fields you’re interested in.
The Aggregation plug-in from ACA IT Solutions allows us to sum up our remaining hours for JIRAs returned by a selected filter. We use it for a grand total for all hours remaining on all SIT JIRAs. We also use this plug-in to display a separate total from the SIT JIRAs returned by our ‘SIT – On Hold’ filter.
The Atlassian Heat Map is used to analyse the functional and status labels for the JIRAs returned by our ‘SIT – All’ filter.
We also use the ‘Two Dimensional Filter Statistics’ plugin from Atlassian to provide an overview of the functionality area of the JIRAs and who they have been assigned to.
The new dashboard gives everyone visibility of the work remaining and makes it easy for testers to come in and pick a JIRA from the ‘SIT Ready to Test’ filter.
General monitoring of the status of all SIT JIRAs is greatly improved. A big bonus is that we can label JIRAs that aren’t currently in the SIT Testing queue, but we know they’re coming (e.g. fixes) so we can include the planned hours of work on them into our dashboard.
Yes, there is a bit of overhead in ensuring that the statuses and remaining hours are updated – but that’s a small price to pay for giving meaning to the SIT Testing queue.