Reality Check Through Scatter Plotting
This post was originally published on Medium and was slightly adjusted when republishing here.
There is a lot of noise in the daily life of companies — especially if you are part of a big corporation. By now, most have undergone at least one transformation and have adopted some form of an agile framework including several tools that come with it — whether useful or not.
Fact is: These tools are being used and if you are one of the lucky ones who works in a proper product-driven company, this article is potentially not as useful.
In this article, I’d like to share examples of how I have seen one particular tool used and how a simple method helped to open discussions that were not possible before. It’s something that I would encourage everyone using the tool to try out for themselves.
The tool I’m talking about is of course:
Story Points
What are Story Points anyway?
According to Atlassian:
Story points are units of measure for expressing an estimate of the overall effort required to fully implement a product backlog item or any other piece of work. Teams assign story points relative to work complexity, the amount of work, and risk or uncertainty. Values are assigned to more effectively break down work into smaller pieces, so they can address uncertainty. Over time, this helps teams understand how much they can achieve in a period of time and builds consensus and commitment to the solution. It may sound counter-intuitive, but this abstraction is helpful because it pushes the team to make tougher decisions around the difficulty of work.
And even though not required by a framework like Scrum, most teams I have come across do use Story Points and some form or Planning Poker to align on the effort that team members think a specific work item will require.
And while this conversation can be useful, in a lot of teams it ends up being fruitless conversations about “Is this 2 or 3 Story Points?”
Other frameworks, like a famous scaling framework, propose the idea of “normalized Story Points” to help predict the story point size for Epics and Features — meaning find a story that requires 1/2 day for coding and 1/2 for testing and call this story your “1 Story Point” work item. Then give every developer or tester in your team 8 Story Points to work on during an iteration and off you go!
The Ugly Mess
When joining companies nowadays and hearing about the organization using Story Points, I have to admit that my alarm bells are going off.
The first thing I do is to check the understanding of what Story Points mean, so I usually ask the open question: “What are Story Points to you?”
What comes next is usually a mixture of all the things stated above and “but we are not taking complexity into account, right?” is being shared every single time in these conversations. Which then usually leads to more proof that the understanding of what is even being done is not aligned within teams and companies.
I usually then try to throw Ron Jeffries into the mix and share that the person who invented Story Points even apologized for it.
I like to say that I may have invented story points, and if I did, I’m sorry now. Let’s explore my current thinking on story points. At least one of us is interested in what I think. — Ron Jeffries, potential inventor of Story Points
But it seems like the arguments and ideas in the article by Jeffries were still not good enough.
Go Where It Hurts
So my next move was usually to see what benefits companies see in using Story Points to estimate OR where they become a challenge — and very quickly you saw an ugly picture of velocities being (mis)-used to predict the future of teams.
Be it for Sprint Planning, the quarterly planning or to predict when an important feature/milestone will be reached.
I have seen a lot:
Teams tweak their velocity because humans know how to trick a system
Complex Excel Sheets in which the Average Base Velocity was calculated and then mapped to the availability of team members
Re-estimating a story during the sprint closing
Weeks spent on getting the capacity and story point estimates into a tool for an entire quarter
But what almost all teams and organizations had in common was that there was some dissatisfaction on the management level about what teams committed to deliver vs. what they delivered.
This usually led to statements like:
Teams need to estimate better
They need to spend more time refining the stories
The preparation week needs to be extended (for quarterly planning)
We need more Agile Coaches
We need more team members
And I could probably go on for even more. At the same time, I was not able to get my message across that these things will not necessarily improve the situation, for different reasons. What I was lacking was a visual system that would give me a seat at the table to discuss what was going wrong.
Scatterplot to the Rescue
Have you tried putting the Story Points on a Scatterplot and mapping Cycle Time to it? - Well, you know….No
This is at least how a conversation between Benjamin Huser-Berta and me must have looked like.
But my answer was honest — I have never even thought about the option of actually reflecting on estimates and mapping them to Cycle Time.
You might wonder, what do you mean by Cycle Time?
Cycle Time: The amount of elapsed time between when a work item started and when a work item finished. - The Kanban Guide
And here is the thing: Using data was not something I was very fond of in the past. I have considered myself as someone who is simply not interested in it and therefore protected myself from ever digging into it — in Coaching we would probably call it “limiting belief”.
But here I was mapping Story Points to Cycle Time and the results I got were: SHOCKING.
You would assume that there is some correlation between story points and time. Not always, as there is variance in complex work. But in general, you would assume that a 13 takes longer than an 8, which takes longer than a 2.
Most of the Scatterplots looked like the ones above, they showcased there was almost no relationship between estimated Story Points and the Cycle time. This was indeed a problem as these organizations used the “normalized story point” approach.
So whenever they did plan their quarter, their firm belief was:
1 Story Point = 1 day of work
Only to find out by the end of the quarter that their delivery rate was again not even close to their desired goal.
As we can see in the 2 Scatterplots above, 2 Story Points could mean a range from a few days of Cycle times up to almost 160 days. Or that a Story estimated with 10 Story Points could take just as much time as a Story with 1,2 or whatever number of Story Points.
This was not a localized problem but something I can see happening for a lot of teams, regardless of the framework they are working as seen in the scatterplot above. A team working with Scrum and wondering why they are not able to finish what they plan to do.
A conversation about how their cycle time (85th percentile) is 20 days or less, allowed us to have a good conversation on why this team is struggling to finish things in a 2-week sprint. A conversation about, what we can learn from the other items (below the red line).
Seat at the Table
And this to me is the real magic and provided me with a valuable lesson for the future: Visualizing something fosters dialogues and conversations about what matters.
Bringing this scatterplot into meetings with management and teams shifted the conversation from:
“They need to estimate better.”
towards
“We might have a systemic issue here.”
With very little effort, we were able to give people visualized information they were able to base their conversations on. It was not information they all individually had in their heads, but it was visible on their screens, in their meeting rooms and finally, a dialogue was made possible.
The visualization helped to understand what was happening in the system and why this department was getting the results they were getting.
We gave them an idea of their reality.
My Recommendation
It’s very tempting in a situation like this to now hire experts and have them work with the team to improve estimates and work on the symptom. Start looking for who is accountable for the situation, change people, change managers, keep everyone busy, and generate a new kind of noise which will probably dilute the issue at hand for a moment.
In a situation like this, my recommendation is and was to start looking into what you already have. Look for patterns that look different, see what you can learn out of them.
Like with the team above — we can see that this team was able to finish most of their work items in 10 days or less. Go and talk to the team — what do they do differently? What can we now do with this information? Is this a concept that would help us with the challenge at hand?
Once you have the conversation, you can bring in different ideas which yes, could include paying closer attention to Flow Metrics and using your Throughput to do Monte Carlo Simulations that will give you a better prediction.
Try It Out Yourself
If you are in an environment in which Story Points are being used to predict the future (Sprint Planning, Quarterly Planning, etc.), take a look at the tools we provide for free.
FlowMetricsCSV
With FlowMetricsCSV you can extract your issues from your work tracking tool and create a Scatterplot with your data.
FlowPulse
With FlowPulse you can automate the process, and instead of extracting the data into CSV, you can directly hook it up with either Jira or Azure DevOps.
If Python is too tricky for you (trust me though, if I was able to do it, you can do it too), you can also do it manually and extract your completed issues from the last quarter into an Excel file-take the Cycle Time and Story Points for your input and create a Scatterplot in Excel.
Using Scatterplots gave and still gives me a lot of answers on why certain things do not work for teams and organizations — i.e. a cycle time of 19 or days or less (85th percentile) for teams that work in a 2-week sprint, says a lot about why a team is not able to finish their tasks within a specific time frame.
As stated above: It gives us a glimpse into reality and opens doors for fruitful conversations.
Use What Makes Sense, But Do Not Lose Sight on Alternatives
Hear me out: I don’t ask anyone to stop doing something that works for anyone. If you are using Story Points (or any other form of estimation) and it works for you and your teams, good job. Keep it going, keep reflecting, and listen to the feedback you get from the inside and the outside.
Also if you read the article and think: What is he even talking about? We do proper product development, we work closely together, work quickly to verify assumptions and then get on with the work that is upcoming and do not even waste our time thinking what happens in 2 weeks, let alone 3 months, I’m happy for you and the work environment you seem to work in.
My reality though is that there is still a big demand for predictability in companies (especially bigger ones) and I’d ask everyone to be curious about other options to predict what can be done and when something can be done.