The Hidden Cost of Noise in Your Delivery Predictions
I’ve been meaning to write about Daniel Kahneman’s work for a while now. I have read his book “Noise” probably for the first time around 2023 but only now more of it makes sense because there is something I pretty much observe in every organization I’ve worked with that maps almost perfectly to what Kahneman describes.
And once you see it, you can’t unsee it.
Let me try to put into words what I mean and only really got to understand since I jumped into the topic of Predictability through my work at LetPeopleWork
The Experiment I Wish I Had Run Earlier
A few years ago, I was working with multiple teams in the same organization. They were all using Jira, all working in similar domains, and all being asked the same question by their stakeholders: When will this be done?
One day, something happened that I didn’t plan but that turned out to be quite eye-opening.
Two teams were independently asked to estimate a set of features that were similar in nature. They didn’t coordinate, they didn’t see each other’s numbers.
And yet, if estimation was a reliable tool, you’d expect the results to be somewhat close, right?
They weren’t. Not even remotely. (Welcome humanity…)
At first, I attributed this to different levels of experience, different contexts — the usual explanations.
But when thinking about Kahneman’s Noise, I realized I had been looking at the wrong thing entirely. The problem wasn’t that the teams were bad at estimating. The problem was something more fundamental.
Bias vs. Noise — And Why It Matters
Most of us working in organizations are familiar with bias.
If you’ve ever used Story Points, Planning Poker or really any kind of estimation, you’ve probably experienced it: teams tend to be optimistic.
They estimate based on best-case scenarios. Kahneman and Tversky called this the planning fallacy, and it has been studied extensively. It’s the idea that we systematically underestimate how long things take.
This is bias. It’s predictable. It goes in one direction. And with enough awareness, you can at least try to account for it.
But here’s the thing that most conversations about estimation miss entirely: noise.
Noise, in Kahneman’s framework, is the variability in judgments that should ideally be the same. It’s not about being wrong in a consistent direction, it’s about being inconsistent. The same person, the same team, the same type of work, give them the same item on a different day, in a different mood, and you get a different answer. Ask a different team altogether and you get yet another answer.
And this isn’t a minor effect. In his research, Kahneman found that noise in professional judgments was often larger than bias. Let that sink in for a moment.
What This Means for Your Roadmap
Here’s why this matters for anyone who builds or depends on a roadmap:
If your delivery predictions are biased, your roadmap is consistently optimistic. You’ll be late, but at least you’ll be late in a somewhat predictable way. You can learn to add buffers, account for the optimism, and manage expectations.
But if your predictions are noisy and trust me they are, then your roadmap is essentially random.
Not just wrong, but unpredictably wrong. Some things will come in faster than expected, others much later, and you won’t be able to tell which is which upfront. The confidence you have in the plan is an illusion.
I’ve seen this play out many times. A team estimates a feature as “about 3 sprints.” Three sprints pass. It’s not done. Someone asks what happened. A very reasonable explanation is provided. But the truth is: the original estimate was never based on data in the first place. It was a gut feeling, shaped by whoever spoke first in the room (hello, anchoring bias), the team’s mood that day, even something as random as the weather and team members being happy about it finally being sunny (writing this blog in grey Swiss winter) or how complex the last feature they worked on was.
That’s not a plan. That’s a coin flip dressed up as a professional judgment (decided to not prompt AI to create a coin dressed up image here - thank me privately).
Try This With Your Own Teams
Here’s something you can try yourself and I’d genuinely encourage you to do it.
Take a set of, say, 10 completed epics or features. Ones that are already done, so you have the actual data on how long they took.
Now, ask two or three people independently without showing them the actual data to estimate how long each of these items “should have taken.”
They know the items, they know the complexity, they were probably involved.
Just ask them to give you their best guess.
Then compare three things:
The variance between the estimators: how different are their numbers from each other? That’s noise.
The average of their estimates vs. the actual: how far off are they on average? That’s bias.
What your historical data actually says: throughput, cycle time, and what a Monte Carlo simulation would have predicted.
In my experience, the results are quite humbling. The gap between what people think happened and what actually happened is significant. And the differences between the estimator people who worked on the same stuff, in the same team, in the same context is often surprising.
This isn’t to embarrass anyone. Quite the opposite. It’s to show that estimation isn’t a skill problem, it’s a human cognition problem.
We’re simply not built for this kind of prediction.
Kahneman spent a lifetime proving that, and he won a Nobel Prize for it.
The Outside View
One of Kahneman’s most powerful concepts is the distinction between the inside view and the outside view.
The inside view is when you look at the specifics of your current situation:
the team composition, the technical complexity, the dependencies and try to reason from there. This is what most estimation sessions do.
The outside view is when you step back and ask:
How long did similar things actually take in the past?
Not in theory, not in a best-case scenario, but in reality.
What does the data say?
Kahneman advocated strongly for what he called reference class forecasting:
making predictions based on the outcomes of similar past situations rather than analyzing the specifics of the current one.
And this is where it gets interesting for me, because that’s exactly what Monte Carlo forecasting does.
You take your team’s historical throughput data: how many items they actually complete per time period and simulate thousands of possible outcomes.
No estimates required. No anchoring. No noise. Just data and probability.
It’s the outside view, automated.
Now, a small but important disclaimer:
Reference Class Forecasting and Monte Carlo simulations are not the same thing. Kahneman's Reference Class Forecasting is about looking at outcomes of a comparable group of past projects or situations, often from outside your own organization to ground your expectations.
Monte Carlo simulation is a statistical technique that uses your own team's historical throughput data to run thousands of random scenarios and produce a probability distribution. The connection is in the mindset: both reject the inside view in favor of actual data.
But Monte Carlo, as we use it, builds the reference class from your own history rather than from external comparisons. It's not a perfect mapping but the underlying principle is the same: stop reasoning from the specifics of this one situation and start learning from what actually happened before.
For more details it might be worth checking out: https://blog.mikebowler.ca/2024/04/10/forecasting-projects/
Where Lighthouse Comes In
I’ll be transparent here this is where Lighthouse, the tool we’ve built at LetPeopleWork, plays a direct role. Lighthouse takes your historical data from Jira, Azure DevOps, or other systems and runs Monte Carlo simulations automatically. It doesn’t ask your team how they feel about a feature (and also does not check the weather forecast)
It looks at what actually happened and tells you what’s likely to happen next.
Is it perfect? No. No forecast is.
But it removes two things that cause the most damage: the noise in human judgment and the bias that comes from our inability to take the outside view naturally.
Instead of asking “How long will this take?” and getting a different answer depending on who you ask and what day it is, you get a probability distribution.
There’s an 85% chance we’ll be done by this date.
There’s a 50% chance we’ll be done by that date.
That’s a fundamentally different kind of conversation.
And in my experience, it leads to fundamentally better decisions.
It’s Not About the Tool
Let me be clear about something, because this is important to me: reading Kahneman’s work has not just changed how I think about forecasting. It has changed how I think about decision-making in organizations altogether and to be quite honest, it is still very much messy and there is lots of untapped potential within companies.
The moment you realize how much noise exists in professional judgment, in hiring decisions, in performance reviews, in strategic planning, in sprint planning, you start to question a lot of things you took for granted. And that’s uncomfortable. And without some help from my co-founder Benji, I would have not gotten to the place where I am now and looking at data when giving an estimate.
But I think it’s also an opportunity. Because once you acknowledge that human judgment is noisy, you can start building systems that help. Not systems that replace human thinking, but systems that inform it. Systems that give professionals the data they need to make good decisions rather than relying on gut feelings alone.
Jim Benson once wrote something that stuck with me:
“Give professionals information they need to be professionals.”
That’s what flow metrics and probabilistic forecasting are about. Not replacing expertise. Making it better.
What I’d Suggest
If any of this resonates with you, here’s what I’d recommend:
Read Noise by Kahneman. It’s not as famous as Thinking, Fast and Slow, but it’s arguably more practical. The chapter on noise audits alone is worth the time.
Then run the experiment I described above with your own teams. See the noise for yourself. You don’t need any special tools for that, just a spreadsheet and some honest conversations.
And if you want to go further, try replacing your next round of estimates with a data-driven forecast. See how it feels to have a conversation based on probabilities rather than gut feelings.
You might find, as I did, that it fundamentally changes the quality of those conversations.
Less theater, more clarity.
Less arguing about whether something is a 5 or an 8, and more focus on what actually matters: delivering value.
If this sounds appealing to you, give it a try. And let me know how it goes.
Hi, I’m Peter, co-founder of LetPeopleWork GmbH and its flagship product Lighthouse, an open-source Monte Carlo forecasting tool for agile teams. I write about flow metrics, organizational development, and the human side of delivery at letpeople.work.