If you’re a stakeholder, a product manager, a department head, a CEO, someone who needs to know when something will be delivered, this is for you. And I mean that genuinely: for you, not at you.

Dear Stakeholder,

I owe you an apology.

Not for what I’m about to tell you, but for what I’ve been telling you for years. What we’ve all been telling you. What this entire industry has been telling you.

We’ve been lying to you. Not maliciously. Not even consciously, most of the time. But lying nonetheless.

Here’s what we’ve been doing. You ask us when something will be done. We go into a room. We talk about it for an hour or two, sometimes longer, it might even take us weeks.
Someone says “feels like a 5.”
Someone else says “I’d say 8.”
We discuss, we negotiate, we converge on a number.
Then we add those numbers up, do some math with a thing called velocity or capacity planning, and out comes a date.

June 12th.

It goes on the roadmap. It goes in the board deck. It goes in the quarterly plan.
Other teams plan their work around it.
Marketing schedules a launch.
Sales starts making promises.
An entire chain of commitments is built on top of that date.

And here’s what that date actually is: it’s the output of a group of people guessing, influenced by whoever spoke first, anchored to whatever number was mentioned earliest, shaped by optimism, and pressured by the unspoken knowledge that saying “we don’t really know” is not an acceptable answer in that room.
Being completely honest with you, it is build on a lot of half-truths and lies.

That’s what we built the plan on. That’s what you built your plan on.
And I’m sorry, because I was in that room, and I didn’t say anything either.

Do you still remember the last time a date we gave you actually held?
I’m asking honestly. Because when I think back, I struggle to find one.
And I’ve been doing this for a long time.

June 12th comes and goes. It’s not done.

There’s a perfectly reasonable explanation, there always is.
A dependency we didn’t see. A technical complexity that only became visible once we started. Someone was sick for two weeks. Public Holidays come out of nowhere. The Summer was unexpectedly nice. Another team didn’t deliver what we needed when we needed it. All true. All legitimate. All completely predictable in hindsight, because these things always happen. They happen on every project, every time. And yet, every time, the estimate is made as if this time they won’t.

The new date is June 26th. Then July 10th. Then “end of July.” Then Q3.

I’ve watched this cycle play out dozens of times. In different companies, different industries, different team sizes. The details change. The pattern doesn’t. And for the longest time, I thought the answer was to get better at estimating. Better techniques, more buffer, better workshops, better spreadsheets.
If we just tried harder, the dates would hold.

They didn’t.

And here’s the part that took me too long to accept: the date was never real.
It was never based on evidence of what was likely to happen.
It was based on a story about what we hoped would happen. And we told that story with such confidence, with specific dates, with decimal points, with colour-coded Gantt charts, that it felt like a fact.
But confidence is not the same as accuracy. And a plan built on unreliable dates is not a plan. It’s a shared fiction that everyone agrees not to question until reality makes it impossible to ignore and that is usually when it is too late.

I know that’s hard to hear. It was hard for me to accept, too. Because it means that all those roadmaps, all those quarterly plans, all those commitments we made with such certainty, they were built on sand. And we all knew it, somewhere, but nobody wanted to be the person to say it out loud.

So here I am, saying it out loud.

The way we’ve been predicting delivery doesn’t work. It has never worked.
And I don’t think trying harder at the same approach will change that, because the problem isn’t effort or skill. It’s that human beings are simply not built for this kind of prediction. We’re too optimistic. We anchor on the first number we hear. And here’s the part that really got to me when I first understood it: we’re inconsistent. The same team, given the same work, on a different day, in a different mood, would give you a completely different estimate. And no amount of planning will fix this.

Your roadmap isn’t just optimistically wrong. It’s randomly wrong. And nobody has been telling you that.

But I’m telling you now. And I’m also telling you that there’s something else we can try.

I’m not going to give you a date anymore. I’m going to give you something like:

“There’s an 85% chance we’ll be done by June 20th. There’s a 50% chance we’ll be done by June 6th.”

I know. It’s not what you’re used to. It doesn’t fit neatly into a Gantt chart. Your board might look at you funny the first time you present it this way. I get it. I really do.

But let me tell you what you’re actually getting.

You’re getting an honest answer. One that doesn’t pretend certainty where there is none. One that shows you the range of what’s likely and lets you decide how much risk to take. If you need to commit to something externally, use the 85% date and sleep well. If you’re planning internally and can absorb some uncertainty, the 50% date gives you an earlier target. That’s not less information. That’s more. That something that helps you make better decisions.

These numbers don’t come from a meeting where people raised their hands and guessed.
They come from our organization’s actual historical data.
How many things we really complete per week.
We run thousands of simulated scenarios based on that, and the result is a probability distribution. It’s not a crystal ball. But it’s dramatically more reliable than what we’ve been doing.

And something changes in the conversations, too.
Instead of “are we on track?” which, let’s be honest, is a yes-or-no trap that nobody wants to answer honestly, the conversation becomes: “how has the probability changed since last week, and what’s driving it?”
That’s a question that leads somewhere. That’s a question that actually helps both of us.

I’m not asking you to lower your expectations.
I’m asking you to raise them, for the quality of information you accept. Because you’ve been settling for false precision dressed up as certainty, and you deserve better than that.

A single date is the delivery prediction equivalent of “trust me.”
A range with probabilities is the equivalent of showing our work.
I think you deserve the latter. I think we both do.

So the next time I give you a range instead of a date, know that it’s not because I don’t have an answer.
It’s because I finally have a better one.
And I respect you too much to keep giving you the old one.

Let’s try this. That’s all I’m asking. Let’s try.

With respect and transparency,

Peter

P.S. Did you know that an 85% confidence level means there’s still a 15% chance we won’t make it?
I know that sounds uncomfortable.
But isn’t it better to know that upfront than to find out on the day it was supposed to be done?

Why I Wrote This

If you’ve read this far, you’re probably either a stakeholder who recognized something uncomfortable in those paragraphs, or someone who’s been trying to have this conversation for a long time. Or you just waited for something insane to happen in this letter.

For the stakeholders: I’ve worked with many of you, and the vast majority don’t actually want a false date.
You want confidence.
You want to know if something is at risk early enough to do something about it. Probabilistic forecasting gives you exactly that, but it requires letting go of the illusion that a single date ever provided it.

For the delivery people: I know this conversation takes courage. It’s scary to say “I’m going to give you a range” when everyone else in the organization is giving dates.
But in my experience, once stakeholders see how much more useful probability-based information is, once they experience a quarter where they weren’t blindsided, they don’t want to go back.

The hard part isn’t the math. The hard part is the first conversation.
This letter is my attempt to make that a little easier.

At LetPeopleWork, this is what we do.
With Lighthouse, our open-source forecasting tool, we make these forecasts possible from your Jira or Azure DevOps data. But the tool is the easy bit.
The harder part is the human part. The conversation part. The part where someone has to look a stakeholder in the eye and say: “The way we’ve been doing this doesn’t work. Here’s something better.”

If you want to try this in your own organization, feel free to share this letter.
Print it out. Put it on someone’s desk. Start the conversation.

And if you want help with what comes after that conversation, we’d love to hear from you.

Peter Zylka-Greger is co-founder of LetPeopleWork GmbH and the co-creator of Lighthouse, an open-source Monte Carlo forecasting tool for agile teams. He writes about flow metrics, organizational development, and the human side of delivery at letpeople.work.

Using this event as an example, we’d like to share in this post how we use Flow Metrics for this part of our Product Management for Lighthouse. We realized that our way of working allows us not only to try out risk-free experiments, but also to react quickly to new developments like a sudden inflow of feedback from new users.

About Value

When we formulate new Features, we try to focus on what problem this will solve. Then we add one or multiple work items below that, which are more concrete steps to achieve this functionality that should solve the problem. Every Feature is assumed to be releasable on its own and to add value for our customers and users. We hope that those Features bring value; however, we can only know this once we have released them and they are in our users’ hands.

Thus, we need to slice a feature as thin as we can. There is no point in spending multiple weeks on something, only to find out it was not what anyone needed. No feature adds value until proven otherwise.

The MCS Backtesting feature was no different; we had no idea if people would care about this. But it was small enough that finding out was worth the risk.

Impact, Confidence, and Ease

When we look at a new feature, we evaluate it through three lenses: Impact, Confidence, and Ease. Rather than debating endlessly or going with the loudest voice in the room, this gives us a structured way to prioritize.

If you want to dive deeper into this approach, Itamar Gilad's writing on ICE scores is a great starting point

The good thing about slicing our features into small pieces is that we usually do not have to spend much time arguing about the impact and the confidence, because they are rather easy to implement.

We can take risks with features we are very unsure about, because they are all easy to implement. Sometimes things will not resonate with the user base as we’d hoped. Sometimes we do something and are highly surprised by the success. If it weren’t for the high “Ease” score, we most likely would never have worked on it. And in turn, we’d never have learned that they are valuable.

MCS Backtesting scored low on Confidence, we genuinely weren't sure anyone would notice it. But the Ease was high, so we shipped it anyway. That's exactly the kind of bet this approach allows you to make.

So what makes a Feature easy for us? Size matters!

Feature Size

It’s rather simple; a small feature is an easy feature. We measure the size of our feature by its work item count. No estimate, no effort, simply the count of work items for a Feature. Below you can see how our features are sized:

You can see that:

• 50% of our Features contain 5 or fewer Child Items

• 85% of our Features contain 7 or fewer Child Items

• Only one Feature is much larger, with 16 Child Items

This is a testament of our effort to aggressively slice our Features to the smallest possible thing that still delivers value. As we mentioned above, this is important because we want to get feedback as fast as possible. We can always do more, but we often shouldn’t. Having such small Features allows us to start and finish our Features, including releasing them, in most cases within a week.

If we look at the Cycle Time of our Features, we can see that 85% of all our Epics are closed within 10 days after we started. 70% are done within 7 days of starting. This means if we start today, we have the Feature not only done, but also tested, documented, and released within a week.

MCS Backtesting was one of those sub-5-item features, that we started and shipped within two days.

The Power of Moving Quickly and Easily

Being able to release Features quickly is one thing. However, we want to gather feedback and adjust our approach based on this. For this, we must ensure that we can react quickly and easily as soon as we get this feedback, which can happen at any point in time.

In our “MCS Backtesting” example, we were surprised by the number of new users this feature has brought into Lighthouse. And with those new users, we got:

• Questions about how certain things work - which were signals for us that the UX can be improved for easier onboarding

• Feedback on existing functionality

• Bug Reports, as our tool was used in new environments

In many organizations, their WIP is abundant. We’re just a two-person company. We can’t afford the luxury of having a high WIP. We release a new version with new features roughly every week. We must be able to react to things we learn.

And we do this simply by:

1. Aggressively controlling WIP

2. Reduce Cycle Time to the minimum

To many people’s surprise, our WIP at the end of a day is often…zero (nada, niente, 0):

As you can see from the above chart, in the last 30 days, there were only three days where something rolled over to a new day. If we start something, the goal is always to finish it on the same day. Because who knows what tomorrow will bring - maybe someone discovered a bug that needs fixing urgently, or we get some feedback we can address right away. Having things in progress will block us from reacting to those things.

When the MCS Backtesting feedback started rolling in, we were ready to act on it. Not because we planned for it (we obviously didn’t), but because we had nothing else blocking the pipeline.

The result of this is a low Cycle Time. From the 57 Stories we closed in this period, only three took longer than a single day (they were closed in two days). So we practically closed everything within the day we started:

This is not happening by accident, but it also drives our decision-making. We also make Stories as small as possible, and we try whenever possible not start something where we know we can’t close it on the same day.

It’s important to mention that proper engineering practices are the fundamental building block: Practicing proper Continuous Integration and Continuous Delivery (see https://minimumcd.org/ for details), keeping the code clean, and having automated tests for most parts of the application will allow you to develop at speed.

By reducing our Cycle Time and aggressively controlling WIP (see below), combined with the thin feature slices, the “Ease” of most of our features is high. Running an experiment is easy and fast. The cost of failure is low. The ability to pivot is high.
Low WIP = Total freedom to pivot. Or how someone once put it: Agile as fuck!

Conclusion

The MCS Backtesting feature is a perfect example of all of this coming together. We built it as a workshop demo, kept it small, shipped it in a week, and woke up to a wave of new users and feedback we hadn't anticipated. Because our WIP was low and our pipeline was clear, we could respond to that feedback almost immediately. No half-finished work blocking the way.

That's the real power of Flow Metrics in product management. It's not just about going fast; it's about staying free to respond to what you learn. Low WIP, short cycle times, and thin feature slices don't just reduce waste. They give you the ability to be genuinely surprised by your users, and actually do something about it.

If you're curious how this could work for your team, check out Lighthouse, our Services, or reach out if you'd like to explore consulting or a workshop.

And once you see it, you can’t unsee it.

Let me try to put into words what I mean and only really got to understand since I jumped into the topic of Predictability through my work at LetPeopleWork

The Experiment I Wish I Had Run Earlier

A few years ago, I was working with multiple teams in the same organization. They were all using Jira, all working in similar domains, and all being asked the same question by their stakeholders: When will this be done?

One day, something happened that I didn’t plan but that turned out to be quite eye-opening.
Two teams were independently asked to estimate a set of features that were similar in nature. They didn’t coordinate, they didn’t see each other’s numbers.
And yet, if estimation was a reliable tool, you’d expect the results to be somewhat close, right?

They weren’t. Not even remotely. (Welcome humanity…)

At first, I attributed this to different levels of experience, different contexts — the usual explanations.
But when thinking about Kahneman’s Noise, I realized I had been looking at the wrong thing entirely. The problem wasn’t that the teams were bad at estimating. The problem was something more fundamental.

Bias vs. Noise — And Why It Matters

Most of us working in organizations are familiar with bias.
If you’ve ever used Story Points, Planning Poker or really any kind of estimation, you’ve probably experienced it: teams tend to be optimistic.
They estimate based on best-case scenarios. Kahneman and Tversky called this the planning fallacy, and it has been studied extensively. It’s the idea that we systematically underestimate how long things take.

This is bias. It’s predictable. It goes in one direction. And with enough awareness, you can at least try to account for it.

But here’s the thing that most conversations about estimation miss entirely: noise.

Noise, in Kahneman’s framework, is the variability in judgments that should ideally be the same. It’s not about being wrong in a consistent direction, it’s about being inconsistent. The same person, the same team, the same type of work, give them the same item on a different day, in a different mood, and you get a different answer. Ask a different team altogether and you get yet another answer.

And this isn’t a minor effect. In his research, Kahneman found that noise in professional judgments was often larger than bias. Let that sink in for a moment.

What This Means for Your Roadmap

Here’s why this matters for anyone who builds or depends on a roadmap:

If your delivery predictions are biased, your roadmap is consistently optimistic. You’ll be late, but at least you’ll be late in a somewhat predictable way. You can learn to add buffers, account for the optimism, and manage expectations.

But if your predictions are noisy and trust me they are, then your roadmap is essentially random.
Not just wrong, but unpredictably wrong. Some things will come in faster than expected, others much later, and you won’t be able to tell which is which upfront. The confidence you have in the plan is an illusion.

I’ve seen this play out many times. A team estimates a feature as “about 3 sprints.” Three sprints pass. It’s not done. Someone asks what happened. A very reasonable explanation is provided. But the truth is: the original estimate was never based on data in the first place. It was a gut feeling, shaped by whoever spoke first in the room (hello, anchoring bias), the team’s mood that day, even something as random as the weather and team members being happy about it finally being sunny (writing this blog in grey Swiss winter) or how complex the last feature they worked on was.

That’s not a plan. That’s a coin flip dressed up as a professional judgment (decided to not prompt AI to create a coin dressed up image here - thank me privately).

Try This With Your Own Teams

Here’s something you can try yourself and I’d genuinely encourage you to do it.

Take a set of, say, 10 completed epics or features. Ones that are already done, so you have the actual data on how long they took.

Now, ask two or three people independently without showing them the actual data to estimate how long each of these items “should have taken.”
They know the items, they know the complexity, they were probably involved.
Just ask them to give you their best guess.

Then compare three things:

1. The variance between the estimators: how different are their numbers from each other? That’s noise.

2. The average of their estimates vs. the actual: how far off are they on average? That’s bias.

3. What your historical data actually says: throughput, cycle time, and what a Monte Carlo simulation would have predicted.

In my experience, the results are quite humbling. The gap between what people think happened and what actually happened is significant. And the differences between the estimator people who worked on the same stuff, in the same team, in the same context is often surprising.

This isn’t to embarrass anyone. Quite the opposite. It’s to show that estimation isn’t a skill problem, it’s a human cognition problem.
We’re simply not built for this kind of prediction.
Kahneman spent a lifetime proving that, and he won a Nobel Prize for it.

The Outside View

One of Kahneman’s most powerful concepts is the distinction between the inside view and the outside view.

The inside view is when you look at the specifics of your current situation:
the team composition, the technical complexity, the dependencies and try to reason from there. This is what most estimation sessions do.

The outside view is when you step back and ask:
How long did similar things actually take in the past?
Not in theory, not in a best-case scenario, but in reality.
What does the data say?

Kahneman advocated strongly for what he called reference class forecasting:
making predictions based on the outcomes of similar past situations rather than analyzing the specifics of the current one.

And this is where it gets interesting for me, because that’s exactly what Monte Carlo forecasting does.
You take your team’s historical throughput data: how many items they actually complete per time period and simulate thousands of possible outcomes.
No estimates required. No anchoring. No noise. Just data and probability.

It’s the outside view, automated.

Now, a small but important disclaimer:
Reference Class Forecasting and Monte Carlo simulations are not the same thing. Kahneman's Reference Class Forecasting is about looking at outcomes of a comparable group of past projects or situations, often from outside your own organization to ground your expectations.
Monte Carlo simulation is a statistical technique that uses your own team's historical throughput data to run thousands of random scenarios and produce a probability distribution. The connection is in the mindset: both reject the inside view in favor of actual data.
But Monte Carlo, as we use it, builds the reference class from your own history rather than from external comparisons. It's not a perfect mapping but the underlying principle is the same: stop reasoning from the specifics of this one situation and start learning from what actually happened before.
For more details it might be worth checking out: https://blog.mikebowler.ca/2024/04/10/forecasting-projects/

Where Lighthouse Comes In

I’ll be transparent here this is where Lighthouse, the tool we’ve built at LetPeopleWork, plays a direct role. Lighthouse takes your historical data from Jira, Azure DevOps, or other systems and runs Monte Carlo simulations automatically. It doesn’t ask your team how they feel about a feature (and also does not check the weather forecast)
It looks at what actually happened and tells you what’s likely to happen next.

Is it perfect? No. No forecast is.
But it removes two things that cause the most damage: the noise in human judgment and the bias that comes from our inability to take the outside view naturally.

Instead of asking “How long will this take?” and getting a different answer depending on who you ask and what day it is, you get a probability distribution.
There’s an 85% chance we’ll be done by this date.
There’s a 50% chance we’ll be done by that date.
That’s a fundamentally different kind of conversation.

And in my experience, it leads to fundamentally better decisions.

It’s Not About the Tool

Let me be clear about something, because this is important to me: reading Kahneman’s work has not just changed how I think about forecasting. It has changed how I think about decision-making in organizations altogether and to be quite honest, it is still very much messy and there is lots of untapped potential within companies.

The moment you realize how much noise exists in professional judgment, in hiring decisions, in performance reviews, in strategic planning, in sprint planning, you start to question a lot of things you took for granted. And that’s uncomfortable. And without some help from my co-founder Benji, I would have not gotten to the place where I am now and looking at data when giving an estimate.

But I think it’s also an opportunity. Because once you acknowledge that human judgment is noisy, you can start building systems that help. Not systems that replace human thinking, but systems that inform it. Systems that give professionals the data they need to make good decisions rather than relying on gut feelings alone.

Jim Benson once wrote something that stuck with me:
“Give professionals information they need to be professionals.”
That’s what flow metrics and probabilistic forecasting are about. Not replacing expertise. Making it better.

What I’d Suggest

If any of this resonates with you, here’s what I’d recommend:

Read Noise by Kahneman. It’s not as famous as Thinking, Fast and Slow, but it’s arguably more practical. The chapter on noise audits alone is worth the time.

Then run the experiment I described above with your own teams. See the noise for yourself. You don’t need any special tools for that, just a spreadsheet and some honest conversations.

And if you want to go further, try replacing your next round of estimates with a data-driven forecast. See how it feels to have a conversation based on probabilities rather than gut feelings.

You might find, as I did, that it fundamentally changes the quality of those conversations.
Less theater, more clarity.
Less arguing about whether something is a 5 or an 8, and more focus on what actually matters: delivering value.

If this sounds appealing to you, give it a try. And let me know how it goes.

Hi, I’m Peter, co-founder of LetPeopleWork GmbH and its flagship product Lighthouse, an open-source Monte Carlo forecasting tool for agile teams. I write about flow metrics, organizational development, and the human side of delivery at letpeople.work.

What started as occasional check-ins turned into the first thing I do every morning. Open Lighthouse, click through the teams, see what’s going on.

No agenda, no checklist, just looking. And yet, this simple routine has fundamentally changed how I work with teams and stakeholders.

In this post, I’d like to walk you through what that actually looks like in practice. Not the theory, not the marketing pitch but rather how I use the tool in my daily work and why it has become indispensable to me.

The Morning Routine

My day starts with Lighthouse open in the browser. I browse through teams and not because someone asked me to, but because it gives me a feel for what’s happening across the system.

I look at work in progress. I check cycle times. I look at the started versus closed work items to see how a team is currently doing. Is it closing more than it’s starting? Or is it the other way around?

These are all pointers I can give to the people I work with. Not answers, but pointers.

And then of course I’m always on the lookout: Is there anything in the system I can see that the team might not have noticed yet? Something worth a conversation?

This part of the routine operates on what I’d call Flight Level 1, the operational level. What is going on inside a specific team, right now?

For example, at the moment I’m looking at a team where the work in progress was drastically reduced, which I think is a very good thing in this case.

Zooming Out: Initiatives and Aging

But I don’t stay at the team level. I might zoom out and look at specific deliveries on an initiative level and with that move more into a Flight Level 2 perspective.

First, I look at the initiatives themselves.
Which ones are aging?
How much are they aging?
Similarly to the team level: how many initiatives are we starting versus how many are we closing? And obviously, how many do we have in progress at the same time?

Lighthouse also gives you a very good indication with the total work item age, which can be an indicator for how stable our system is.

And then I go into the details. Depending on the service level expectation, I might filter for the initiatives that are close to it or the ones that are already above it.
Is there anything we can do? Is there any support that could be needed?

The intent is always the same: start finishing, stop starting. Which sounds so easy but is incredibly hard to achieve depending on the complexity of your system and organization.

Forecasting: From Gut Feeling to Probability

Another thing I use Lighthouse for and this might be the part that changed my work the most is specific forecasts.

A product owner asks me: “Can you help to forecast when this feature will be done?” Instead of guessing or pulling a date out of thin air or asking some silly questions, I open Lighthouse and look at the forecast. I might play around with it a little. Check the team’s metrics. If more teams are involved, I look at those too.
The question I always like ask myself: does this make sense?

But it goes further than individual features. It might be something for a release where I know a date is planned. Maybe the release even has dependencies, some companies use gates.
With Lighthouse, I can set up a delivery, set the delivery date, and Lighthouse will give me a forecast of the probability of all that is planned actually happening. It tells me whether my date is something we could actually reach or something that might be a bit difficult to reach.

This is just data. What we do with it is up to us.

But here’s where it gets really valuable: I go through the list and see which epics, features, whatever you call them in your context have a very low probability of being completed until that date. And I see this early. Not on the due date itself but weeks before.

This sparks useful conversations with the responsible people. What does it mean for us? Do we need to reduce scope? Can we remove the epic? Are there dependencies? Are there other things we can deprioritize?

The idea is to have the first conversations weeks before the due date when there actually is still time to discuss, converge, and actually do something about it.
Rather than finding out on date X that we missed it and say “Bad Luck - will for sure work out next time”

And the nice thing is: there is nothing you have to do manually. Lighthouse updates these forecasts on a regular basis. Even if you just sit down every Monday and check how things are progressing, whether anything changed, there is nothing you have to do except clicking into the right window.

That takes a lot of complexity out of it. I don’t have to sit down and calculate anything myself which apparently is a fear for a lot of people hearing about Monte Carlo Simulations.

The Power of Specifics

Of course, it also works the other way around. If there is a need for a very specific forecast say, we have one epic and we know we need to finish it by the 8th of March — Lighthouse can tell me how probable that is.

I go into the tool and ask:
How probable is it that we will finish 25 items until the 8th of March? Lighthouse tells me the likelihood is less than 7%. Which is not a lot.

But it doesn’t stop there. It also tells me: based on what we have at the moment, the 85th percentile suggests the 23th of March. And on the other hand, if the 8th of March is the date that isn’t movable, it tells me how many items I can realistically expect:
5,8,11 or 14 depending on my risk appetite.

That’s a conversation I can have with a stakeholder. Not “we’ll try our best” but “here is what the data says, and here are our options.”

Building Trust: Forecast Backtesting

One of the great things we’ve introduced recently is forecast backtesting.

And to me, this is really a feature that I like to use to give people confidence.

Because still, my experience is that forecasting feels spooky and sci-fi to a lot of people. It’s just not what we’re used to. So a feature that lets you prove the forecast is trustworthy? That’s incredibly valuable and hopefully gives organizations trust in this.

For example, I ran a backtest recently for a two-week time period in October. The 85th percentile said 12 work items would be finished. The actual throughput was 16 work items. That gives me confidence, it’s not completely off. And I can also play around with the historical windows. Change it to 14 days and suddenly we’re very close to the 70th percentile, which was 16 items.

This proved to be very fruitful for building trust with the people I work with. It turns the abstract into something tangible and verifiable.

Monthly Reports and the Role of Data

I’ve also started using Lighthouse for monthly reports and not in a sense where I do a reporting and people are scared of me sending the report, but rather I go through the data and compile suggestions, observations on what can be done, where patterns are emerging, where something might need attention.

But here’s the thing I always emphasize: Lighthouse provides you the data. The context, the real, messy, human context lives in the teams and the people working with it. They will always have more information than any dashboard can show.

I like to say: the data is a pointer. A pointer to ask: Is this accurate? Does it reflect the way we work at the moment? Is there anything we can do?

And if we do change something, the data comes back and says: yes, it worked. Because we can see something moving.

Why This Matters: Evidence Over Gut Feeling

What I’ve experienced with teams is that they get genuinely enthusiastic whenever they see a change they made reflected in the data. It’s like a booster.

A lot of times when we change something in teams or on an organizational level, let’s not ignore the other flight levels here we trust our gut and say “yeah, it feels better.” And that’s a nice indicator. But I think it’s a lot stronger when we actually have evidence. When we can see that we’re moving more things through the system. When we can see that cycle time came down because we did something specific.

Same on a Flight Level 2 when starting to right-size initiatives and you can see that cycle time is actually going down, thats a big booster on a different level.
We get feedback quicker, features out to customers faster, and therefore learn sooner.

This is something that is very motivating. We all know that change is difficult and tiring. But having something solid, actual proof that what you did made a difference? That keeps people going.

The Short Version

If I had to summarize it: there is not a single day where I’m not using Lighthouse. There is always something to find. Whether it’s a team whose WIP just dropped, an initiative that’s aging beyond its service level, a forecast that needs updating, or a backtest that helps me build trust with a stakeholder.

You don’t need a lot of data. Installation is easy.

It’s a tool that changed how I have conversations. Not because it gives me answers, but because it gives me the right questions.

And in my experience, that’s worth more.

Meaning: roadmaps that the team can actually deliver in the time available.

In my environment, budget is allocated on a yearly basis. That means my roadmap must realistically cover at least one year. You can’t rely on wishful thinking, oversized ambitions, or inflated capacity assumptions. You need something grounded, data-driven, and operationally realistic.

Over the years, I’ve refined a workflow that reliably gets me to a roadmap the team can commit to. It’s a mix of probabilistic forecasting, lean decision-making, and practical collaboration with the people who will actually deliver the work.

If you struggle with unrealistic roadmaps, slipping commitments, or annual budget pressures, I hope this workflow helps bring clarity, predictability, and sanity back into your planning process.

1. Start With a Prioritized List of Epics

Before thinking about time or capacity, start with outcomes.

What do we want to achieve this year? What matters most for our customers and for the business?

This results in a prioritised list of epics, ranked by value and impact.

The roadmap will only ever be as good as this list.

2. Establish a Realistic User Story Capacity

Next, I need to know how much work the team can realistically deliver in one year.

I use Monte Carlo simulation to answer a simple question: “How many user stories can the team complete in a year with 85% confidence?”

Example: The simulation tells us the team should be able to complete 230 user stories or more with 85% confidence.

That number is now my upper bound for planning.

3. Decide How Much Capacity Goes to Feature Development

Not all capacity goes into new features. The real world also includes:

• Bugs

• Technical debt

• …

So I consciously decide how much of the yearly capacity I want to allocate to feature development.

Example: even if the team can do ~230 stories, I might allocate only 200 to planned features.

The rest is buffer — for reality.

This prevents overcommitting, protects the team, and keeps the roadmap sane.

4. Create a Quick High-Level Plan

At this point, as a team, we haven’t reviewed epics deeply yet. I don’t want to waste the team’s time on detailed estimations before I know which work is likely to make the cut.

So I use historical data to create a quick sizing rule of thumb: 85% of the epics we’ve delivered in the past required 30 user stories or less.

Therefore, as a placeholder, I temporarily assume 30 stories per epic.

Using our example:

• Capacity for features = 200 stories

• Average epic placeholder size = 30 stories

• High-level capacity = ~6 epics

Those 6 become the initial high-level plan.
This step is extremely lightweight — and that’s intentional.

5. Review Only the Selected Epics With the Core Team

Now that I know which epics might fit, it’s time to review them together with key players: architect, lead developers, lead tester, cybersecurity expert, …

This review takes time, so we make sure to apply it only to the most important items, those likely to make the cut!

This is where depth meets focus.

6. T-Shirt Sizing and Risk Assessment

In the review, the team assigns:

• A T-shirt size (XS/S/M/L/XL)

• A risk level (low / medium / high)

T-shirt sizes are then translated into the number of expected user stories, but with a twist. Instead of arbitrary numbers, k-means clustering algorithm on historical epic sizes is used to to determine how many user stories should be used for each T-shirt size. E.g. T-shirt size M typically means 18 stories

This keeps estimates consistent with our actual delivery patterns.

7. Replace the Placeholder Epic Sizes With Human Estimates

Once the review is done, the high-level plan becomes more accurate: instead of “every epic is 30 stories”, we now have epic-specific estimates grounded in historical data and expert judgment.

This hybrid approach keeps estimation cost low while ensuring quality where it matters.

8. Refine the Plan Based on Updated Estimates

We can now re-check the plan against the capacity.

Two scenarios:

• The epics do not fit the “200 user story capacity”: I remove the lowest-priority items until the scope fits.

• The epics fit, with room left over: review few more items (step 5).

This is the stage where realism strengthens.

9. Replace T-Shirt Estimates With Actual User Stories

As we break epics down into user stories (just-in-time refinement):

• the T-shirt estimates are replaced with the actual story counts.

• The forecast updates automatically (point 10)

This makes the roadmap both adaptive and transparent.

10. Continuous Forecasting

At any point in time, the Monte Carlo simulation provides a continuously updated, data-driven answer to a simple question: “When will we finish the planned scope?”

That planned scope is not a single, fixed thing. It is composed of two parts:

• Epics that have already been analysed, where we have a reasonable understanding of how many user stories are required.

• Epics that have not yet been analysed, where the work is still represented by forecasted user stories derived from historical data.

At any moment, the roadmap can be understood as a mix of actual user stories and forecasted user stories.

Early in the journey, uncertainty dominates. Most of the scope exists as forecast rather than concrete work: classic fog of war. But as execution progresses, epics are refined just in time, forecasted user stories are replaced by real ones, and uncertainty steadily decreases.

What starts as a range of possibilities gradually collapses into reality.

The forecast evolves with the work, not despite it, allowing the roadmap to remain both realistic and responsive throughout execution.

Why should you try this approach?

1. Fast initial planning

You can produce an initial plan in hours, not weeks.

2. Just-in-time work

The team invests effort only into the epics that really matter. Waste minimised. Lean, for real.

3. Human input + Data synergy

The final plan is:

• based on team judgment

• backed by historical performance

• reinforced by statistical forecasting

Teams commit much more easily to plans they helped shape.

4. Avoiding the trap of estimating everything

Estimating everything is waste. Because half of them will never happen. Or they’ll mutate by the time we get to them.

In Conclusion

There are many ways to plan and track a team’s work.

This one is mine, but only until I find a better one.

There are also many tools that can support this way of working.
The one I use is Lighthouse.

Stephanie Leue described a dynamic I’ve seen many times:

This sparked a lot of thoughts and I immediately went into writing about this.

The result is that overcommitment becomes normalized.

This doesn’t happen because people don’t care.

It happens because three forces converge: psychology, culture, and systems.

1. Psychology

Several human tendencies make overcommitment feel natural:

• Fear of rejection and conflict. Saying yes avoids discomfort in the moment, even if it creates bigger problems later. But the later part takes the burden away from us and we can still hope for that wonder to happen.

• Optimism bias. We underestimate complexity and overestimate capacity. A project that “should only take a month” often drags on for months or even years. But the next time we estimate, it definitely will be different (Spoiler: It won’t)

• Escalation of commitment. Once effort is invested, stopping feels like failure. We double down. And we think back on all the conversations we would need to have to reverse it.

• Short-term relief. Agreeing resolves tension now, while declining creates immediate friction people prefer to avoid, especially if they are overloaded already.

Each of these biases creates drag. Together they form a gravitational pull toward commitments we cannot realistically keep.

2. Culture

The surrounding culture determines whether these tendencies get amplified or balanced:

• Output over outcomes. When success is measured by volume of delivery, more is always seen as better—even when it creates little value.

• Roadmaps as contracts. Tentative plans harden into promises the moment they’re shared. Changing them later feels politically dangerous and we are being told from early childhood that we should avoid anything that is dangerous.

• Lack of psychological safety. In many environments, raising concerns is not rewarded. People learn that silence is safer than dissent. And this does not even need to be rational and based on the organization they now work in but rather something they bring along from their past.

In such cultures, overcommitment isn’t a mistake. It’s a survival tactic.

3. Systems

The structure of work itself reinforces the default of yes:

• Incentives misaligned with value. Funding and recognition are often tied to the size of an initiative. The bigger the budget or headcount, the more important the work appears. This encourages teams to inflate scope to secure resources, rather than narrowing focus to deliver real value.

• High work in progress. Starting more feels productive, but excessive WIP slows everything, hides bottlenecks, and erodes trust in delivery. At the same time it creates the illusion that we are actually working on all these things (Try Henrik Kniberg’s name-game to see that you are actually not working on all these things)

• Leadership unable to say no. Customer escalations, competitive moves, or stakeholder demands often get accepted without trade-offs. Teams learn resistance rarely changes anything.

When the system rewards saying yes, individuals adapt accordingly.

The Cost

Overcommitment is like starting a race with weights strapped to your ankles. (Just had to think about a race as the Ironman Switzerland is going on outside my window)

You’re moving, you’re expending energy, but the drag is built in from the beginning.
The harder you push, the heavier it feels.

The costs pile up:

• Focus is scattered.

• Delivery slows instead of speeds up.

• Technical debt and rework accumulate.

• Burnout becomes normalized.

• Trust between teams and leadership steadily erodes.

In time, people stop believing promises at all. Both leaders and teams lose confidence in each other.

What Can Be Done

The antidote is not to demand more discipline or tougher commitments.
It is to change the conditions so that saying no is safer and smarter than saying yes.

Some practical shifts:

1. Measure outcomes, not just output.
   Value should be tracked by problems solved and impact delivered, not by volume of features shipped.

2. Keep plans adaptable.
   Treat roadmaps as hypotheses that evolve with new evidence. Avoid locking in promises that cannot survive contact with reality.

3. Make it safe to push back.
   Leaders can model this by showing vulnerability, asking for feedback, and treating dissent as ownership.

4. Limit WIP and finish what is started.
   Reduce parallel commitments so work flows faster, bottlenecks surface earlier, and predictability improves.

5. Model saying no at the top.
   If leaders cannot decline requests or defer work, teams will not feel safe doing so either.

6. Create evidence that supports pushback.
   This is where tools like Lighthouse come in. By showing probabilistic forecasts, cycle times, and WIP trends, teams can point to data instead of opinions. Pushback becomes less about courage and more about evidence. When delivery ranges are visible, the myth of “we can do it all” is harder to sustain.

7. Reward learning and finishing.
   Celebrate outcomes achieved, insights gained, and improvements made to the system. Over time, this builds a new norm where sustainable focus replaces constant overload.

When these conditions are in place, the gravitational pull shifts.
Overcommitment stops being the default.
Pushing back becomes the responsible, expected thing to do.

The Open Question

Pushing back will always require some courage. But courage alone is not enough.
It needs the right environment, supportive systems, and leaders who invite it.

So the real question is:
How do we create organizations where effective pushback is encouraged before overcommitment becomes the norm?

Pancakes

This morning, Costanza and I found ourselves deep in a discussion about flow. Not at work, not in a meeting room — but while waiting for pancakes and omelettes at the Jackson Lake Lodge diner.

We had woken up at 4:00 AM to photograph Mormon Row at sunrise (yes, totally worth it), and by 7:00 AM, we were back indoors, shivering slightly, sipping coffee, and waiting for breakfast. The moment felt still—until our shared obsession with continuous improvement kicked in.

So naturally, we started mapping out the diner’s operations through the lens of flow.

The diner offered both takeaway and sit-in service, and that sparked a string of questions:

• Should takeaway orders be prioritized over dine-in ones?

• Should there be two separate flows (and teams) for these order types?

• Or should the focus be on minimizing order cycle time, without distinguishing between order classes?

Translating Breakfast into Kanban Terms

As we sipped our coffee, we couldn’t help but think in Kanban terms: expedite lanes, classes of service, cycle time, throughput.

From a pure flow perspective, reducing cycle time sounds like the universal goal. But, of course, any business has constraints: staffing, cost, space, and the ever-present need to be profitable. You can’t just throw more people at the problem.

That’s when the key question hit us — the one that underpins any flow discussion, whether in manufacturing or software:

What are the service level expectations required by the business?

The Role of Perceived Wait Time

From a business standpoint, you don’t necessarily need to serve orders as fast as technically possible. You need to serve them fast enough that the customer experience remains positive.

But “fast enough” is a slippery concept — it depends heavily on context and customer expectations. When I’m seated, coffee in hand, and engaged in a good conversation, my tolerance for waiting is quite high. I’m not watching the clock. But if I were a takeaway customer, standing near the counter, glancing at my phone, every minute would feel like five.

So here’s the insight we landed on before the omelette arrived:

Customer expectations aren’t uniform. And your flow should be designed around meeting different service level expectations, not blindly optimizing speed across the board.

For example:

• Takeaway customers might expect their order in under 5 minutes — with a goal to hit that for 75% of cases.

• Sit-in customers might be perfectly happy if their food arrives within 15 minutes — let’s say 85% of the time.

This nuanced thinking helps shift the conversation from “how fast can we deliver?” to “how do we deliver fast enough, sustainably, and where it matters?”

Wrapping Up (Before the Pancakes Get Cold)

This wasn’t a business meeting. Just two friends — one from engineering, the other from software — talking flow over breakfast. But the lesson is real:

Whether you’re running a diner, a factory, or a product development team, flow optimization doesn’t exist in a vacuum. It should always be grounded in customer experience and service level expectations.

Now, if you’ll excuse us, our omelettes just arrived — and even continuous improvement takes a pause for pancakes.

*If you are interested in Lorenzo’s pictures, you can check some of them out at https://thedistantview.myportfolio.com/albums

In this post, I’d like to share my perspective on the usage of AI for developing a software product. I’ll share my actual experience with it, rather than making bold claims.

Who is this for?

If you check out LinkedIn (among other places), it’s nearly impossible to not read about all the things AI will change, who it will be putting out of a job, and more…

It feels like suddenly everyone is an AI expert, Vibe Code Coach, or Prompt specialist.

I’m not opposed to embracing new technologies. It’s what agility is also about. The integration of AI will certainly change how we work (hopefully for the better). With this post, I want to give an insight into how AI can be used for developing a software product.

If you are not developing yourself, this may give you a good idea of what’s possible nowadays. If you do develop, maybe it will encourage you to give it a try as well.

There may be better tools and better approaches to what I did. I don’t claim you should do the same. My goal is to share my experience, not to sell you a specific method.

My Background

While I’ve studied Software Engineering, and was working the first years of my career very closely in this domain, I’ve been focused more on the people and processes in the last years. Writing code was not part of my daily professional life anymore.

While I know some things about a Frontend, I’ve always liked the Backend and automation work way more. Frontends were a “necessary evil” more than something I liked to do.

Needless to say, this showed in the first version of Lighthouse:

A Backend Developer in a React World

It was clear to me that this was not something we should leave as is. So, after having it on my to-do list for years, I finally had enough motivation to dive into learning React.

As much of a culture shock as New York was for Sting, React felt very strange to me initially.

I started out learning the “old school” way:

• Watching Tutorial

• Doing online courses

• Browsing the web for specific help

I’d like to think that it was very useful for me to learn the basics like this:

• What is a component?

• How should I structure my app?

• How can I write tests?

• What’s a state, and what are props again?

But I also realized that with such an established technology, Large Language Models (LLMs) such as ChatGPT are very helpful.

The following is a preview of how Lighthouse looked in one of the first iterations where we switched to the new React-based frontend:

At this point, I was mainly using ChatGPT to help me with specific problems or errors. This was way faster than having to sift through 20 search results from Google or asking a question on Stack Overflow (with the added benefit that ChatGPT was not complaining about how I wrote my question…).

And while it was helpful, it didn’t feel so nice yet. You always had to context-switch between a browser window and the Integrated Development Environment (IDE).

IDE Integration

That’s when I realized that there is a quite nice integration into VS Code for GitHub Copilot. This extension would allow you to chat and ask questions, where it’s directly having the context of your current file (or other files you’re referencing). And not only can you ask questions, but it is also possible that the LLM would directly make changes in your files.

This integration felt much more natural, removing context-switching, making it as easy to use as, for example, built-in refactoring tools your IDE is offering.

This was not just speeding up general development; it also allowed me to apply AI for more specific tasks where I needed more context. I used this extensively to improve the testability of my components, refactoring them and increasing the test coverage. This was hard to do without having the context, as it potentially affects many files in the project.

It also helped build up end-to-end testing using Playwright. I’ve never used Playwright before, but similar to React, the LLMs helped me get up to speed very quick, and it was one of the smoothest experiences I've had learning a new technology.

Curious in how we test Lighthouse? Read the dedicated Post about this!

This was nice, and I was happy. Things were working smoothly. I didn’t think there was much to improve. And then Agent Mode happened…

Agent Mode

In early 2025, Microsoft released an update to GitHub Copilot and its VS Code integration, so that now “Agent Mode” is supported. Instead of “just” asking questions or doing “simple edits”, Copilot was able to act as an agent, figuring stuff out on its own, looking for more context, and iterating on its own changes (for example, if tests are failing).

The demos were impressive, and my first testing also revealed that it seems to hold up on its promise. Far from perfect, but very capable, and it sped up development even more. It felt like it was supercharging my development efforts.

It’s been a year since I got my hands dirty with React, and I was doing ok. That said, I don’t see how I ever spend lots of time to make something really “nice”. It’s just not that interesting to me. But I thought, why not let the Agent mode of Copilot have a go at it? Out of curiosity, I threw the following prompt at it and let it work:

Copilot was at it for about 60-90 minutes. While some things weren’t working yet, it managed to change the full application and implement my requests. The new UI looked like this:

It took me about a day to clean everything up, make sure all tests were working again, and have everything deployed. One day, to fully rework the UI. For me, the person who does not really care about UIs. We could try to estimate how long this would have taken me to implement without the agent mode, but realistically, it would simply never have happened.

While ChatGPT and the “regular” Ask and Edit modes of Copilot helped me speed up, Agent mode made things possible that I would not have thought I could do:

Vibe Coding?

You may wonder, so is using this Agent mode what people call Vibe Coding? In my understanding, it’s a clear no.

I still try to look at the code, understand what’s going on, and sometimes manually adjust it. I don’t trust the output, and it’s my responsibility as a developer to make sure whatever I commit into version control is of good quality.

Following is the “definition” of vibe coding from Andrej Karpathy, who first mentioned the term on Twitter beginning of 2025:

So as long as you still inspect the code, you’re not “vibing”. AI-assisted coding, even if it’s doing the bulk of writing the code, is not vibe coding, if you’re still checking it out yourself, and potentially fixing some problems based on the resulting code.

Why do I care about that? I’ve seen the term being used extensively online (especially LinkedIn), where I had the feeling that many people don’t fully understand what it means. So I thought it’s good to clarify this.

This doesn’t mean that you can’t build something in a “Vibe Coding” mode. But for now, at least personally, I wouldn’t trust this enough to build a product, unless it’s something quite simple.

Conclusion

Is AI extremely helpful in building software? Yes! Can I build my software faster than before? Yes! Is it allowing me to build things that would not have been possible without it? Also yes!

You can see, I’m quite a big fan. Looking at Lighthouse as a case study, you can see how this can change a product. All of it could have been possible without AI, but realistically, I needed help from outside. Using LLMs and agents integrated with your IDE is upgrading your skill as an individual developer, allowing you to go faster and further!

I’m convinced the role of software engineers is going to change (if it hasn’t already). For me, embracing those new tools has been a game-changer. And while it’s not producing results that are always up to my quality standards, it has allowed me to become a “one-person development team”, covering all skills needed to develop a product. That’s pretty cool!

While you don't have to understand the details of what's going on to use Lighthouse, we recommend eventually going through it. There are things that help predictability in general that become more clear when you understand how MCS works, while Lighthouse is based on certain assumptions that will impact the results.

Note that this is the state as of February 2025 - the most up-to-date way Lighthouse is forecasting can always be found in the docs itself, as this post may be outdated.

Monte Carlo Simulations Explained

A forecast using Monte Carlo Simulation will simulate a period using a historical Throughput. You can use it to forecast how many items you can manage to close in a specific amount of time, or how long it will take to get x items done.

Monte Carlo Simulations assume that your future will look like your past. This is great because in the past you most likely had sick days, holidays, people leaving the team, people joining the team, etc., so you don’t need to do special “Capacity Planning” or anything of those sorts.

Note that this also means that if your future will not be very similar to your past, MCS is maybe not the right approach (but after a couple of weeks you will have the new data to get started with).

Be aware that this is not magic, and you assume that “your future looks like your past”. Meaning that it highly depends on the input in the form of Throughput we feed it with.

We recommend that you dive deeper into MCS by reading our blog post An Introduction and Step-by-Step Guide to Monte Carlo Simulations and follow along the exercise in a spreadsheet.

About Predictability

Now that you know how MCS works, and before we dive into the details of Lighthouse, let’s briefly explore Predictability in the context of MCS.

You are more predictable the closer your forecasts are together. Meaning if there is only a small gap between your percentiles (for example 50/70/85/95%), you would be more predictable.

Let’s look at an example with two teams for this and forecast how many items we can get done in the next 14 days.

==== Forecast Team A ====
*How many Items can be done in 14 days?*

95% Chance: 3 items or more

85% Chance: 5 items or more

70% Chance: 12 items or more

50% Chance: 27 items or more 

==== Forecast Team B ====  

*How many Items can be done in 14 days?*

95% Chance: 9 items or more

85% Chance: 12 items or more

70% Chance: 14 items or more

50% Chance: 19 items or more 

According to our definition, Team B is more predictable. Team A might have a higher upside, but the chance of that happening is not that high. The 70% chance is very similar for both teams, but if you want a higher probability, it drops very low with Team A. If you need to get 9 items done in 14 days, you would trust Team B more to get the job done as they are more predictable.

Predictability does not equal speed. If you get one item done consistently every week, you are very predictable. You might also be slow, but those are two different things. We recommend first focusing on predictability, and once you are happy with it, focus on what you can do to improve speed.

Improving Predictability

If you want to improve predictability, you should focus on Flow. Flow Metrics are a great indicator and starting point to do so. Keep your Work Item Age low, act on things that start to get older, and the rest will fall in place.

We recommend reading our blog post Using Flow Metrics to get an introduction to Flow Metrics.

How Lighthouse Forecasts

Now that we’ve covered the basics, let’s look at how Lighthouse is forecasting. We’ll use an example to explain the interesting parts. We’ll use some scenarios to explain how Lighthouse is running the forecast, starting from the most simple examples to more complicated ones.

We’ll look into how Lighthouse is running When forecasts for projects. The How Many and When on Team level are the basic operations that are also covered in the above-mentioned blog post.

Teams, Features, and Projects

Forecasts will always be made for specific Teams. The forecast will calculate When a certain number of Work Items will be done. Those Work Items are automatically fetched based on the Team Backlog (see Team Work Item Query) and belong to a specific Feature of a Project (based on the Project Work Item Query).

Lighthouse is only aware of Work Items if they are linked to a project. If your team is working on 5 Features, but only 3 of them are linked to Projects in Lighthouse, the other 2 Features and all the work items that belong to them will be ignored, and the forecast assumes you will only work on the 3 Features that are known. It is therefore important to make sure you have all the features that are relevant to you covered within a Lighthouse Project.

If Project 2 is not defined in Lighthouse, Work Item 2, 4, and 5 will also not be known to Lighthouse.

Why the Backlog Order Matters

When Lighthouse is forecasting, it will always take all the Features of all Projects into account. If you set up a new Project, you want to know when this will be done taking into account all the other things that are going on. You can’t make a forecast for a Project in isolation, but you will always look at the full known backlog.

That means, if you really want to ignore everything else, just remove all other Projects. That’s not recommended though.

Because Lighthouse works this way, this is also the reason why the order is so important. In the above image, you can see that Feature 1 is ordered above Feature 2, which itself is above Feature 3. That means that Lighthouse assumes we first focus all our efforts on Feature 1 (assuming a Feature WIP of 1), then on Feature 2, and only then on Feature 3. If you forecast Project 2, the forecast will take into account that Feature 1 and Feature 2 must be done before that.

The Feature View in the Teams Detail Page will show you all the Features the team is contributing towards in their order.

The Impact of Feature WIP

By default, Lighthouse will assume that your Teams will work on one Feature at a time (Feature WIP = 1). In other words, this means that the forecast will check how many items can be done, and assume all those items will contribute to the first feature. So the Features will be done in sequence:

If we increase the Feature WIP to 2, we will assume that the work that is completed is distributed to two Features (as long as we have 2):

The same pattern can be applied with an even higher Feature WIP (as an example 3):

In simple terms this means, that the higher the Feature WIP, the more focus goes away from the highest priority item (Feature 1) and the earlier we start on less important things (Feature 2 and following). So instead of getting the most important thing early, we get everything later. This contradicts our ambition to continuously deliver value in small batches.

We highly recommend keeping your Feature WIP as low as possible. Lighthouse can help you make the effects of a high Feature WIP transparent. It also is able to automatically adjust the Feature WIP based on your work in progress. See the Team Settings for more details.

Example Scenarios

So let’s go through some scenarios to look at what happens when Lighthouse forecasts.

For simplicity, we’ll talk only about Teams and Features. As you learned above, this is tied to Projects. However, when it comes to forecasting we just care about the Features in Order, independent of which Projects they are in. To simplify this a bit, we’ll skip the Projects in the examples.

1 Team - 1 Feature

The most basic case is if we have a single team, working on a single Feature. This equals a simple When forecast. We take the remaining Work Items for this Feature (all items that belong to this Feature that are Not Done).

Then we calculate how long it will take to complete all those items.

Feature WIP does not matter in this situation, as we’ll only have one Feature available. Thus all the work goes into that Feature.

1 Team - 2 Features

If we have a single team, but 2 features, it starts to get more interesting. Lighthouse will first get all remaining work items for each Feature.

Then it will simulate a single day, and reduce the remaining work from the Features till there is nothing left. Once both Features have no remaining items left, the simulation is done. For each feature, we’ll count the simulated number of days it took.

For Feature 2 we will not start with the 1st day, but the count will resume from where it was when we finished Feature 1.

This is done 10’000 times and the results are stored (how long did it take till the Features were done).

If the Feature WIP is 1, all simulated Throughput goes into the first Feature, until it’s done. From then on, all simulated Throughput goes towards the second one.

If you have 1 item left for a Feature, and you simulate 2 or more items to be done that day, the full Throughput will go to Feature 1. Let’s book this under context switching cost.

If we have a Feature WIP of 2, we simulate the Throughput each day. Then we randomly pick which Feature to assign this to.

Before you ask, it’s not possible to somehow weigh Features, for example, saying we work 80% on Feature 1 and 20% on Feature 2. If you work on multiple things, items will be randomly distributed.

2 Teams - 2 Features

If we have two teams working on two features independently, it’s the same case as in 1 Team - 1 Feature. We simply do it for both teams individually.

2 Teams - 1 Feature

A tricky case is if we have a feature that is being worked on by two teams. This means both teams have work to do for this feature. Lighthouse will run two forecasts, one for each team with the remaining work for the Feature (similar to 1 Team - 1 Feature). Then both forecasts will be stored for this Feature. Lighthouse is then presenting you the forecast that is predicted to be done later.

That does not mean that one team is slower, they might just have other work that is ordered higher up.

What Lighthouse does in such cases is not ideal. It suggests that there is a 95/85/70/50% probability for the forecast that is predicted to be done later. However, as it’s two forecasts that both need to happen, the real probabilities would be 90/72/49/25%. This would be even worse if three or more teams would be involved. Dependencies are horrible, try to get rid of them. While we know it’s not properly done at the moment, we are not sure what’s the best way to handle this. We’re open to ideas.

Dependencies

Lighthouse does not offer any possibility to define dependencies in the sense of:

• This work item can only be started once that work is done

• This Feature has to wait for the completion of those two other Features before we can start

Lighthouse will also not offer this going forward.

This is for two reasons. First, it’s a lot of work to do. It may sound easy, but it will complicate the whole tool. And while we are not opposing investing in the tool, we only want to do it when we think it makes sense. Which brings us to the reason #2: We don’t like dependencies. While there may be some, you should try to aggressively remove them as well as possible. As long as we have lots of dependencies, a tool like Lighthouse will not help you very much.

Please have a look at the work of Prateek Singh, who wrote among others, this very interesting blog post: Understanding the Impact of Dependencies

Conclusion

Congrats, you made it to the end of this page! We understand that this may have been the opposite of light reading. As mentioned in the beginning, you don’t need to know all of this when you start out with Lighthouse. But as you go on, you may wonder why certain things work how they do. And sometimes the answer can be found in the way Lighthouse works.

You may wonder why this is so complicated. It would be a lot easier if there were no dependencies between teams, teams would only focus on one feature at a time and could deliver projects without other teams. Also, it would be nice if we could keep a proper order in our backlog. We just try to help you cope with reality…don’t hate the player, hate the game. Or better yet, change how your organization plays the game.

You can see the end result of our documentation at https://docs.lighthouse.letpeople.work:

Documentation Experience

Up to now, I’ve experienced two kinds of documentation:

1. User-facing documentation, most often written in a dedicated tool like Word or dedicated Content Management Tools

2. Documentation aimed at developers, most often written in markup languages (like Markdown), for example as part of a README or Wiki

In my experience, developers often dread the former, as it’s connected with something that you just have to do, not something you want to do. You have to use those annoying tools (that are very different from what you normally use day-to-day), and due to that, we try to delay this as much as possible. So when we finally can’t shift it anymore, we don’t remember the details which makes it even more of a slog and just reinforces the belief that this is annoying work.

“Developer Documentation” usually flows better. It may not be everyone’s favorite task, but it’s often less of a pain to do this.

My assumption is this is because you can stay within the tools you are used to, and the documentation work is often done much closer in time since the change that is being documented happened.

The whole experience feels different when I modify a Markdown file, commit the changes, push them to a repository, and the change is instantly there.

With this in mind, we tackled the User Documentation for Lighthouse as well. But before we get there, let’s quickly talk about one of my favorite topics: Automation.

Automation Automation Automation

I love automation. It’s one of the coolest feelings if you can automate something that was a cumbersome manual task before (especially if it had to be done repeatedly). It’s right up there with getting your tests to pass.

When I recently was writing my CV, I was putting the following in the About Me section:

This was supported by the fact that the whole CV itself is written in LaTeX, and a CI pipeline is creating a PDF out of this and publishing it. Instead of a time-stamp, the CV is versioned (well the version is based on the last change date, but still).

What I’m trying to tell you is that I’m serious about automation. Apart from me enjoying to do it, automation will lead to many improvements:

• It democratizes knowledge and makes it available to anyone. We don’t depend on a single person remembering how to do a thing, even if that person is yourself (I certainly don’t remember that LaTeX command to compile my CV…)

• You make a process repeatable

• Ideally, you also make the process versionable (for example by storing all the things needed to run the automation under version control)

• You remove waste, as you don’t need to spend time doing that thing manually and instead can spend it on other things (like automating more things!)

  • Note that just because something can be automated, doesn’t mean it’s worth it. There is always some cost to create the automation, which may be a lot higher than what you save.

As Lighthouse is something I’m spending my evenings and weekends on, and we concluded that “proper documentation” may be worth having, I combined it with doing something fun and automating as much as I possibly could. But before we get that, let’s first dive into why we concluded that the documentation was worth having.

Why Bother To Create A Documentation?

Initially, we had all the documentation on GitHub itself, as part of the README.me. This works but also expects that users are willing to navigate to your GitHub project and browse through the Readme. The experience is not ideal, and not everybody is a developer and wants to deal with GitHub (which looks scary with all those files and badges and pipelines, etc…). You also lack a decent search (for the documentation itself) as well as some elements to structure a document.

As more people got onboarded to the tool, we noticed that similar questions were popping up, and some would have been answered in our Readme already. Which is an indication that people do not read through that.

We took this as a feedback and decided to invest time into a documentation that:

• Is well structured (or at least could be well structured)

• Is easy to write and extend

• Can be searched

• We could host online

• We could automate the heck out of (maybe this was just my requirement :-)

Read on to learn what we did in detail to achieve all of the above.

Documentation as Code

Based on my previous experience, I aimed at a solution that was done in Markdown (or similar). That would allow us to keep using the same editors as when we change code, and we could have the documentation in version control. In fact, we can keep it in the same repository as the application itself.

As we were using GitHub to store our code (did we mention that Lighthouse is free and Open-Source?), it made sense to use a cool feature from GitHub for the documentation: GitHub Pages.

GitHub Pages allows you to host a website directly from your GitHub repository. Push some changes, trigger a CI Pipeline, and within a minute, the new version is published. And all of this is based on markdown files in your repo.

Two cool features from GitHub pages are:

• You can set up custom domains, so instead of a default URL, it can be anything you want it to be (assuming you own the domain). This allowed us to host our documentation under https://docs.lighthouse.letpeople.work, even though it’s hosted on GitHub itself.

• GitHub supports Jekyll, which allows you to use various themes to create static sites based on your markdown files.

Just The Docs

One of the available themes for Jekyll is Just The Docs. As the name suggests, this is a theme that helps you write nice documentation. It’s free to use, very simple yet powerful, and needless to say, the documentation of the theme is great.

Jekyll is based on Ruby and you can run it with any template easily on your local machine (for example before you set up your CI Pipelines on GitHub).

Once done, you can work with Markdown as you are (most likely) used to. In Lighthouse, we keep our documentation within the docs folder and create subfolders for each area that we display in the navigation.

While Jekyll respectively Just the Docs add some configuration options (for example the frontmatter on top of the file to describe the page details and navigation), you can mainly use the regular markdown syntax that you know (and love). Create links to other pages, use # for your headings, add tables where they make sense, and insert pictures to visualize what you’re describing. Which brings us to the topic of screenshots.

Screenshots!

It was not long after I started writing that I thought I should add a screenshot. Easy thing, just let me spin up the app and capture one. Store the file in my docs/assets folder, reference it, and it works.

Except, what if the app changes. I obviously should retake the screenshot. But do I remember that? And it would be nice if all the screenshots would follow a similar pattern (for example use the same team/project names). I could sense that either I’m automating this right away, or we will have not updated screenshots very quickly (automation = fun, taking screenshots = boring).

In Lighthouse, we run end-to-end tests using Playwright. This is a great tool for testing, but the fact that it’s spinning up the app automatically, putting it in a known state, and can be run from our pipelines, made it also interesting for taking screenshots.

If you write your testing framework in a decent way (which you really should), you can reuse all your page objects for the sake of taking screenshots. Then very quickly you get simple tests that you can use for recapturing screenshots. To take the screenshot you see above in the Feature Page, this code is used:

Without going too much into the details, the test uses a tag (@screenshot) and then spins up the application, puts it into a known state (updating Teams and Projects), navigates to the Overview Page, and takes the screenshot.

As we can run Lighthouse in docker, we can spin up a container to test against (to make sure we don’t have any leftovers of some other testing we did):

The argument --grep @screenshot will make sure we only run tests with the @screenshot tag.

And our takePageScreenshot method also checks if the image is different than the one we currently have, and will only save it if that’s the case (with some threshold). That way, we don’t spam our git history with minor changes (for example if just the date is updated).

Pretty cool stuff so far, but to complete it, we want to run all of this automatically.

Pipelines!

The final piece of the puzzle is the Continuous Integration and Deployment. Lighthouse has three CI Pipelines:

• The main pipeline is building, testing, packaging, and deploying Lighthouse.

• The docs CI Pipeline is triggered if anything within the docs subdirectory is changed. This will redeploy the Jekyll page and deploy it to GitHub pages.

• The screenshot pipeline, which can be triggered manually (and then runs against the latest dev docker image), also is triggered if any new release is created (and then runs against this specific version).

The screenshot pipeline is running the Playwright tests that create the screenshots (if they changed). On top of that, it will create a Pull Request so a human can manually review the changes:

If the pipeline is rerun, the PR will be updated with newer screenshots (just in case something needs a fix). Once merged, the docs pipeline will kick in (as we changed something within the docs folder) and the latest screenshots will appear in our online documentation within the next 1-2 minutes.

Conclusion

Not that you needed any evidence, but I think this post is proof that I really like to automate stuff. At the same time, I did notice how much easier it is to write user documentation in Markdown. It just feels more natural than using some other WYSIWYG editor. I hope I could inspire you to also think about how you write your documentation, I can highly recommend this approach!

In this post, I’d like to explore how Lighthouse is tested and try to explain what purpose the different tests serve.

Why Bother With Tests?

You may wonder, if this used to be a personal project, why was I doing tests from the very beginning? The reason is simple: I enjoy writing tests. I don’t see this as a burden. I think it’s a great feeling if you wrote a test for a new feature, implemented the functionality, and now this and all other tests are green.

Or if you just reproduced a bug through an automated test. Yes, just reproducing the bug via test is already a great feeling. Because that means we have a clear indication that it’s fixed (when the test passes). And that this specific problem won’t happen again. Ever.

In my experience, many developers dislike tests. At best, they accept that they have to write them. Instead of embracing what great tests allow you to do (change your code with confidence!), they add them on top. Because someone expects the code coverage to be at least <add random number here>. I’m aware that’s by far not true for many developers and companies, but it’s something that I have experienced more often than developers being excited about tests…

Can I Trust This Change?

Building up a decent test suite, which covers various aspects of the Lighthouse functionality has been instrumental in evolving the application over time. There have been several bigger refactorings, but also the occasional bug that slips through and is discovered later.

Knowing that you have a set of tests that will make sure you will not break anything that used to work is very calming. It allows me to make aggressive changes, refactor as I see fit, and occasionally try something, with very limited risk.

Interestingly enough, it’s often not during refactoring that tests start to fail. But it has happened several times that I changed something and a seemingly unrelated test suddenly failed. I would be lying if often my first thought would not be: “Oh, I probably need to adjust the test so it passes again”, contemplating what I could change to make the test more reliable. Only to then find out that I just unintentionally broke something.

Can We Trust This Release?

As a user of Lighthouse (if you’re not yet - head over to https://letpeople.work/lighthouse to learn more!), you also get something out of this. Maybe you just don’t want to deal with amateur software, that continuously breaks. One release something works, another time this is suddenly broken. Once an application gets a reputation for being unreliable, it’s hard to get rid of it.

What’s more, because Lighthouse is free and open-source, many people associate this with “unprofessional”. While many open-source projects are just hobby projects, probably a lot of open-source is done more professionally than many paid software applications and services. As our target audience with Lighthouse is also corporations (or people within corporations that want to make a change in how they forecast), it’s an advantage that we can show that things are properly tested. On many levels. Continuously.

So let’s dive deeper into this.

What Do We Test?

Lighthouse has unit-, integration- and end-to-end tests. They are all run continuously; every time a change is pushed, it’s run. And if a single test fails somewhere along the pipeline, we immediately know something broke.

Backend

The Lighthouse Backend is built in ASP.NET Core in C#. Most classes were written using TDD and therefore are unit-tested, leading to a high code coverage (not that this tells us that much). You can also see this for yourself on SonarQube Cloud.

In addition to the Unit Tests, also some integration tests are run. As Lighthouse works with data coming from both Jira and Azure DevOps, there is a hard dependency on those services. And while this can be mocked away to test most things, we test that those services also return the expected data, by having set up demo projects for each and making sure we get what we expect.

Frontend

The front end is built in React, and we’re using vitest for testing. In general, every component should be covered by tests. Compared to the backend, the coverage is not as good (see SonarQube Cloud). The main reason for this is that I had no prior experience with React, so a lot of focus was simply getting it to run, sometimes creating suboptimal code that was harder to test. That said, as we continue to add features, fix bugs, and refactor in the process, the coverage has increased over the last months.

End-To-End Tests

For quite a while, the Unit- and Integration tests for the back- and the front end was all there was. However, we noticed at one point that, while still everything is green, sometimes things break because the individual units were working, but put together something was still off.

So we added end-to-end UI Tests using Playwright. This allows us to spin up the whole application (assuming the unit- and integration tests were running ok), and test end-to-end scenarios. Those tests interact via the Browser with the Frontend and work against the real Backend, which in turn gets data from a real Jira and Azure DevOps instance.

Using some neat features like fixtures, we can make sure each test runs in isolation (so it does not depend on other tests and can be run in any order), while also setting up Lighthouse into a known state (for example by making sure certain Teams or Projects are added).

Note that we always prefer to run unit or integration tests. End-to-end tests are expensive because they are slow to run. We try to make sure the basic scenarios work but don’t try to cover everything.

Deploy The Demo App

If all other tests have been successful, we are then deploying the app as part of the CI Pipeline to our demo environment. This is not an automated test, but it helps to check if the app behaves as we would expect in the demo environment with the sample data.

Publish A Docker Image

In parallel with the deployment of the Demo App, docker images with the tag dev-latest are created and pushed to the GitHub Container Registry.

After the package is pushed, the end-to-end tests are run against this docker image. This makes sure we do not have any different behavior between the regular package and the one that is deployed in docker.

The automatic deployment of the Demo App as well as the docker image also simplifies the manual verification. Getting the latest version is just one click or one command away, so there is not really an excuse to not verify each individual change.

Conclusion

Only if all the above-mentioned tests were run, do we consider releasing a new package. The tests give us a lot of confidence to do this often, and history has shown that we can trust our test suite to tell us if we just broke something.

Testing has been an integral part of Lighthouse from its first commits, and we’re happy we’ve invested in this early on, everything else would be unprofessional.

Note that this is the state as of February 2025. The most up-to-date way Lighthouse is forecasting can always be found in the docs themselves, as this post may be outdated.

If you look for the basic features, you can find it in this post:

In this post, we’ll share some advanced (but awesome) features of Lighthouse that make your life better, and your forecast more realistic, and hopefully help you drive meaningful change in your organization:

1. Default Feature Sizes

2. Unparented Work Items

3. Feature WIP

4. Active Feature Visualization

Default Feature Size

Lighthouse Project Forecasting is based on the child items of your Features (for example your Epics or Feature work item types). However, not every Feature will be broken down already. This is great because it might not be worth the effort. However, when we run a forecast, we need to know the approximate size. Lighthouse offers functionality to define a Default Feature Size that is applied to features that don’t have any child items yet.

The default feature size can be defined either as a fixed number, using historical data, or using an estimate.

Lighthouse will determine the size in this order:

1. Number of Child Items read from your Work Tracking System

2. Estimation of Size per Feature

3. Default/Historical Feature Size

If no child items are defined, it will fall back to the estimation. If no field is defined, or the field is defined but no estimate is found for a feature, it will fall back to the default or historical value (depending on your configuration).

Default Number of Items

The easiest way is to define a default number of Items, for example, 15 items. You can base this number on your historical data or just go with your gut. The value is fixed and will be applied to all Features that qualify for using the default value.

Historical Feature Size

Instead of a default size, you can also choose to let Lighthouse calculate your historical feature size. If you do so, you have to define a percentile and a query.

As an example, if you chose the 85th percentile, it means that it uses the size that 85% of your Features had. If you go for 50, it uses the size that at least 50% of your features had. This is more flexible than using a fixed number, as Lighthouse will recalculate this value for you when updating. So over time, the system learns and the values will automatically adjust.

Estimated Size

On top of the default size, which will be similar for every Feature, you can also specify a field that would include an estimate. This allows you to use the default size for features you have not looked at at all while providing more details for some things that you may have started to refine, but have no child items yet. Simply specify the name of the field that contains your estimate, and Lighthouse will extract the data from your work tracking system.

State Override

Sometimes we may have child items already, but we are still in the process of refining them. So instead of using the 3 child items that are already there, we may still want to use the estimate (which may say 12 for a feature) for some time. In order to achieve this, you can define which states should ignore the actual child items. For those states, the real number will always be ignored, and the estimate or default size will be used.

Visual Indication

You will get a visual indication for every item that is using the default/estimated size. That helps you identify where you have pending work to refine the features. This way Lighthouse allows you to create realistic forecasts early on, while also making sure you are not forgetting about the fact that some features still need to be refined (which may change your forecasts).

Unparented Work Items

Sometimes (or shall we say “in the real world”) there is work that does not belong to a specific Feature. Still, it needs to get done. These may be small improvements that were planned (or promised) or some bug fixes.

What Lighthouse offers instead is to collect all work items that are not having a featured parent into a “virtual” unparented feature that is also taken into the forecast. All you need to do for this is to create a query that searches for the work items in the individual team backlogs and Lighthouse will group all of them together and show them as “Unparented Items” for your projects.

The query will only include items that are not already part of this project (under a ‘regular feature’). If we look at the screenshot above, it means that Lighthouse will go through the Backlogs of all the four involved teams, check every item that has the label My Release, and will add it to the virtual unparented feature if it’s not part of a “regular” Feature.

Feature WIP

If your team is working on multiple Features at the same time, Lighthouse allows you to define this number, as this will impact your forecasts for projects, and will lead to different predicted delivery times.

As an example, if you work on a single feature at a time, this feature will be done as fast as possible. If you work on two features, the first feature will finish later (as at least some of the teams effort goes to the second feature). In an ideal world, you have a Feature WIP of one. Your reality might look different, and that’s ok. Just know that ideally you should strive to be as close as possible to a Feature WIP of one.

Automatically Adjust Feature WIP

Probably you don’t want to go and count your Feature WIP and adjust the Lighthouse settings every few days. So instead, Lighthouse allows you to automatically adjust the Feature WIP with every refresh to the number of Features that are right now being worked on by this team.

Visualize The Impact of Feature WIP

Your Feature WIP might not always be the same (again, the real world is messy). You can set your Feature WIP to anything you like, there does not need to be a correlation to what is actually happening (although obviously it would be good if the number reflects reality…). Lighthouse allows you to easily adjust the Feature WIP to visualize the impact. Once adjusted, it will automatically reforecast the completion dates and likelihoods for each milestone:

You should strive for a Feature WIP that is as low as possible, ideally 1 or 2. However, if your reality looks different, it makes more sense to set Lighthouse up accordingly, as otherwise the forecasts will be off. You may use Lighthouse to make it transparent what a change in Feature WIP can mean in terms of Feature delivery, which could be a good conversation starter to make a change.

Active Feature Visualization

For both Teams and Projects, Lighthouse will visually indicate what Features are actively being worked on.

This is a great indicator to see if we are focusing on the most important items (as Lighthouse is based on the order in your Work Tracking System).

On a team level, you can see how many Features are actively being worked on, and if they are “at the top” of the list (if not, this may be a good conversation starter).

For projects you get the same view, but with all the involved teams. In the following screenshot you can see that we currently work on the second feature, while the first one is not even refined yet (indicated by the use of the Default Size).

Getting Started

Now that you know about what sets Lighthouse apart, you may wonder how you can start to use it. Our documentation at https://docs.lighthouse.letpeople.work is the best place to get started. Do you need support? Join our Slack Channel and get help from us and other users who already use the tool. Or reach out to us to discuss what options we offer to businesses that need tailored support.

Note that this is the state as of February 2025. The most up-to-date way Lighthouse is forecasting can always be found in the docs themselves, as this post may be outdated.

What Is Lighthouse?

Lighthouse is a tool that helps you run probabilistic forecasts using Monte Carlo Simulations in a continuous and simple way. It connects to your Work Tracking Tool (currently Jira and Azure DevOps are supported) and will automatically update your team’s Throughput and your project’s forecasted delivery dates.

You can use it with a single team for doing manual “When” and “How Many” forecasts, as well as for tracking projects with one or multiple teams.

Free, Open-Source, and No Cloud Connection

Lighthouse is hosted on GitHub under an MIT License. It is free to use, whether you are an individual, small company, or big enterprise. No hidden fees, no subscription service. Furthermore, Lighthouse is running on your systems. There is no cloud service involved where it stores any data, everything stays on the system that runs it.

Cross Platform

No matter what system you have, Lighthouse will run. We support Windows, macOS, and Linux, and provide Docker containers to run it.

Check out the Installation portion of our documentation for more information.

Jira and Azure DevOps

The two most popular systems for tracking your work are supported. There is a seamless integration with both Jira and Azure DevOps, and you get the full power of their respective query languages to define your teams and projects.

Check out the Concepts portion of our documentation to learn more.

Live Demo

If you want to see it live, you can either just run it in docker, or even simpler, check out our Live Demo.

Continuous Forecasting

One of the main features of Lighthouse is the fact that all your forecasts are automatically updated. Forecasting is most powerful if you do it continuously, and Lighthouse was designed with this in mind. Set up your teams and projects, and always get an up-to-date view of your forecasts.

Team Level Forecasts

Simply define your Teams and start running forecasts based on the Throughput. This eliminates the need for Story Points or other forms of estimations and accompanying meetings, allows teams to focus on delivering value while providing stakeholders with accurate insights, and also supports you in creating a realistic plan for your Sprints, Product Increments, etc.

Project Tracking

Create a new Project by telling which teams are involved and what Epics or Features are part of it, and you are ready to go. Instead of days, an initial project plan is done in minutes. And kept up-to-date as long as the project goes on:

With Lighthouse, you can manage projects from initial planning to delivery, whether you have a single or multiple teams that contribute to features.

Forecast Sprints, Product Increments, Releases, Quarters, or anything else that you define as a Project. Lighthouse helps you create transparency and is based on the historical data of the involved teams.

Have some important date coming up? No problem, dynamically create Milestones and see how likely it is to deliver each Feature of your project till then.

Keep the Overview

Lighthouse offers you one dashboard for all your Projects. You can see real-time updates reflecting the current state. This helps you focus on analysis and action, rather than gathering data and spending time in status meetings.

Getting Started

Our documentation at https://docs.lighthouse.letpeople.work is the best place to get started. Do you need support? Join our Slack Channel and get help from us and other users who already use the tool. Or reach out to us to discuss what options we offer to businesses that need tailored support.

Consultants and American Football - How It All Started

Yes, you’ve read that right. Two main reasons why Lighthouse was created were the statements of an external consultant on one hand and American Football on the other.

In January 2024 we were working with a consultancy that was engaged to help us deliver a new product. During a discussion about Quality, Practices, and Continuous Integration, a lead developer said:

It doesn’t make sense to have green pipelines. We’re in the development phase, so the tests can’t always be working. This is also why we should use Gitflow to keep the main branch stable.

I was puzzled at such a statement, especially for a greenfield project using state-of-the-art technology. My inner developer was shocked and angry: That’s not how we are supposed to do things in 2024. However, after some time, I was wondering whether I might just be out of touch. I have not been writing much code in recent years (apart from a few scripts), so who am I to think I know better. Don’t get me wrong, I was still convinced that the statement was not correct, but I wanted to gain some practical experience again instead of just assuming things.

A Highly Competitive Idiot

Before I elaborate on what the National Football League has to do with the creation of Lighthouse, a quick note on myself. I’d describe myself as a highly competitive idiot. Everyone who will work with me for some time will know what I mean (and that I mean this most positively) with the idiot part. I like to have fun and not take things too seriously (frantically points at all the memes).

What many might not see behind the memes is that I’m ultra-competitive. If you tell me that I can’t do something, you can be sure I will invest a lot of time to prove you wrong (yes, it’s that easy to get me to do something). I don’t see the point of aiming for mediocracy (“Well it’s not great, but the others are not better either”). I do understand we can’t be the best in everything, but it’s no excuse for just accepting mediocre or subpar practices.

The great thing about working in the Software industry is that the bar keeps moving, what was state of the art 5 years ago, might be seen as outdated today. This also means that once we get to state-of-the-art, and we keep improving, we get to define state-of-the-art.

Now if someone comes and sells me something outdated 10 years ago (hello Gitflow…) as the thing we should certainly be doing, I get triggered…

American Football

If you are following the NFL, you are probably aware that in the first weeks of February, the Super Bowl is happening. This ends the season, and it will take till September until the new one begins. The Super Bowl in 2024 happened on the 11th of February, just after I had the conversation described above.

This is important because this meant that suddenly my Sunday evenings were free (in Europe, most NFL games happen on Sunday evenings). Instead of watching games, I have time to spend on writing code. And I could do this in the way I wanted to.

Do Things My Way

The cool thing about personal projects is that you can do whatever you want, however you want. My goal was, apart from the actual product that was built, to make sure I could refresh my knowledge of various engineering practices, and (re-) gain practical experience. Specifically, I wanted to do:

• Use Test Driven Development as good as I could

• Do Continuous Integration and Continuous Deployment

Some proof for that is visible below, as the first commits show that one of the first things I did was to add CI Pipelines, integrate with SonarCloud to check the code, and add tests:

I tried to use those practices wherever they made sense. And the more the project grew, the happier I was that I invested in proper engineering practices. While going into detail on all the practices and how they helped me develop deserves its own blog post, the conclusion I can draw after one year is that it’s certainly not impossible to keep the pipeline green.

Make Forecasting Accessible

Next to my personal reasons for starting this project (remember, I basically just wanted to prove that certain engineering practices work), another driver was to make the topic of forecasting a bit more accessible. In the weeks before, I did a post on How Monte Carlo Simulations work, which got some attention. I wrote a basic tool in Python to run simulations based on CSV data, but thought that there could be something more accessible and feature rich.

A 15% Forecasting Solution

While there were various tools, from spreadsheets to full-fledged Cloud services that integrated into Jira and Azure DevOps, I had the feeling that something was missing. You either get very basic functionality, or you have to pay quite some money for and/or need some high-level access to your systems (to install new plugins in Jira/Azure DevOps).

At Let People Work, we discussed this and concluded that there is a gap for a forecasting tool that would allow someone in a big corporation to try out this way of forecasting. It should be a 15% solution, something that can be tried out without the need to ask for money or permission from anyone. For this, the tool would need to be:

• Free to use

• Open-Source (in case internal IT catches wind of an unauthorized tool they can go and check the code if they want)

• Run locally (no need to upload to some cloud service that you may not trust)

While we aim to build up services around the tool, what we develop shall remain available for the people we had in mind initially.

What Have We Learned

To wrap up this post, I want to share some of the learnings that we had within the first year of Lighthouse's existence.

Technology and Practices

I am a Software Engineer by trait, but in my professional developer life, I mainly worked on Desktop Applications (.Net/C#/WPF) before in 2019 I switched towards a full-time Scrum Master role. So Lighthouse was a chance to catch up on all the fancy tech and practices. The following things were new to me (meaning while I knew those things, I haven’t built something proper beyond a “Hello-World” with them) and I learned over the last year while building Lighthouse:

• ASP.NET Core

• Razor Pages

• ChatGPT (to write code)

• Docker

• React

• vitest

• Playwright

• GitHub Actions

• Jekyll

• Just-The-Docs

The engineering practices (TDD, CI/CD) were crucial, as several major refactorings happened within the first year. Having an extensive test suite made this possible without breaking (too many) things in the process.

Product Management

Apart from the technology side of things, we also started to learn things about product management. As people started to use it, we got feedback and feature requests. As the tool is free to use, we’re spending our free time on it. So as time goes on, we learn about how to best manage stakeholders, we experiment with how we can scale up (maybe having 5 different communication channels isn’t the smartest idea…), and what features to really follow up on and what to drop (saying no to things is a lot easier if you work on it on the weekends).

Conclusion

Lighthouse started as a hobby project for me, trying to catch up with technology and engineering practices. This is something I would recommend to everyone, instead of just claiming some random stuff about how AI will change the industry on LinkedIn, go actually build something and learn while doing it.

And as we head into Super Bowl LIX (go Eagles!), this also means our time will free up again. So try out Lighthouse, join our Slack Channel, and help us make your forecasting experience better!

Read on to learn more about Little’s Law, what it is good for (and what it is not), and how we can practically apply it.

What’s Little’s Law?

Little’s Law is named after John Little, an American researcher. He was working in the field of Queueing Theory. Little’s Law is an equation, which takes the following form:

Where L is the queue length (or Work In Progress), λ is the “arrival rate”, and W is the Wait Time (or Cycle Time). While the above equation was published by Philip Morse, Little was the first to prove that the law is valid.

Kanban and Flow Metrics

When we’re applying Kanban (as defined by the Kanban Guide), we have four measures of Flow:

• Work In Progress (WIP)

• Throughput

• Cycle Time

• Work Item Age

With those four metrics, we can measure the flow of value through a system. As you can see, the original form of Little’s Law includes WIP (L) and Cycle Time (W) but not the arrival rate. However, there is another form of Little’s Law that looks at Throughput (or departure rate).

What’s Little Got To Do With It?

When we use departure instead of arrival rate, the equation looks like this:

This puts three of our Flow Metrics in relation to each other, which means we cannot look at one of those three metrics in isolation but have to understand that if one changes, the other two will be affected as well. On the other hand, if we want to adjust one, we can look at how we can change one (or both) of the others to achieve the desired change.

Be Aware

Before we look in more detail at what the Law is good for and how to apply it, a few words of caution.

First and foremost, the law (especially in the form of Throughput/Departure Rate) has a set of assumptions that must be fulfilled for it to be valid. Daniel Vacanti wrote an excellent article about this if you want to dive deeper. The summary is that we expect a relatively stable system, where items that are started are also finished and we’re not accumulating more and more WIP over time.

Furthermore, the law talks about averages of the metrics over some time. In many posts that reference the law, this fact is either implied or simply skipped or forgotten. That fact makes the law not well suited for forecasting the future…

What Is It Good For?

You might be inclined to “plug in your numbers” into the equation and forecast the future. For example, you might think, that to halve your Cycle Time, you can simply half your WIP while keeping your Throughput stable.

While I understand the thought, there are several problems with it:

• The law applies to historical data, not to future data.

• You will not be able to tell how your Throughput is affected by this change.

• The law is based on averages, so it falls victim to the Flaw of Averages

So if we can’t use it for making forecasts about the future, what is it good for?

Note: You might also realize that if you plug your historical numbers into the formula, it will not be “correct”. This is because most likely one (or more) assumptions of this form of the equation are not fulfilled. I highly recommend reading the series from Daniel Vacanti on Little’s Law for more details.

Instead of forecasting the future, we can use Little’s Law to identify improvements. It helps us formulate hypotheses and craft experiments that aim to change our Flow Metrics in a given way. The nice thing about this is, that we have the data to see if our experiments worked as we expected. We see how this could work with a concrete example, but first, let’s do a bit of elementary school math.

Solving For X

We already saw the most used form of Little’s Law above:

But we can also adjust it, so we solve it for either average Throughput or average WIP:

Depending on which Flow Metric you are most interested in adjusting, you can pick the most useful form.

A Little Less Conversation…

…a little more action, let’s dive into how we could apply this knowledge. Imagine this:

Let’s split this exercise into two parts, before we design an experiment, let’s first think about our Flow Metrics.

Impact on Flow Metrics

Let’s bring in the equation with Average Throughput on the left side of the equals sign:

The throughput depends on WIP and Cycle Time. If we want to increase our Throughput, we have the following options (in no particular order):

• Decreasing Cycle Time while keeping WIP stable (or increasing)

• Increase WIP while keeping Cycle Time stable (or decreasing)

• Decrease WIP while decreasing Cycle Time more than you decrease WIP

• Increase Cycle Time while increasing WIP more than you increase Cycle Time

Note that I skipped writing “Average” for the sake of readability.

You notice that I always wrote the change for both, Cycle Time and WIP. As a change in one will most likely affect both other metrics, I prefer writing down what I’d like to achieve for both metrics on “the right side” of the equation.

Another thing that you might be wondering about: I could increase WIP and that might lead to a positive change. But I always hear we should limit and lower WIP?

Indeed, if you can manage the effect the increased WIP has on your Cycle Time, that might be a good strategy. This could happen if you are performing below the “optimum” of your system, and have too much slack. An example could be that you are “starving” the system, so once you close an item, there is no item to be pulled, leading to wasted time for the team. An increased WIP might be useful in such a scenario.

You can see such a scenario in one of our case studies about Controlling WIP.

In any case, Little’s Law does not tell us what is “good or bad”, it just shows the relation and gives us the above options if we want to increase our Throughput. So let’s try to put this to good use.

Designing Experiments

We can now think about various ways to achieve the change in Cycle Time or WIP to increase Throughput. Let’s look at two examples:

Decreasing Cycle Time while keeping WIP stable

What the above basically means is to keep the same amount of items in progress, but get them done faster. One way to do this is to look at how you slice your items. Can they be sliced into smaller pieces? Depending on your workflow, this might lead to faster feedback, so you discover potential problems earlier (think about integration issues or early customer feedback). This in turn will support you in getting the items closed faster, in turn decreasing your Cycle Time while keeping WIP the same.

Increase WIP while keeping Cycle Time stable

If you are happy with your Cycle Time, you might consider increasing your WIP and see what happens. Whatever mechanism you use for controlling your WIP, think about allowing a bit more work into your process. Now you might not want to overdo it, and it should still be controlled, but it doesn’t mean you're not supposed to change it ever.

While the result can be an increase in Cycle Time, it might also help you achieve faster flow. The good thing is that we have the data to check this.

Evaluating your Hypothesis

Whether you call it hypothesis, improvement action, retrospective outcome, or anything else. Ultimately it’s just an assumption that when we do X —> Y will happen. We formulated our experiments based on Little’s Law, and we can very easily measure if reality matches what we hoped expected would happen.

If yes, great! You’ve successfully applied Little’s Law. You may give me a shout-out and share your experiment!

If not, great! You can see what happened instead of what you thought, and based on this either abandon or adjust your experiment. You may give me a shout-out and share your experiment and learnings!

Little’s Law is not magic, it will not solve your problems. But it can be really helpful in identifying what you could do to adjust your process and achieve a better flow.

What’s My Age Again?

The attentive reader might have noticed that we only focused on three flow metrics, but there are four of them. What about Work Item Age in all of this? Is this not relevant?

You have correctly spotted that Work Item Age (WIA) is not part of the equation of Little’s Law. However, it is of great relevance anyway. It is part of the assumptions (check the resources for more details on this). WIA is a leading indicator for Cycle Time. While Throughput and Cycle Time are lagging indicators, WIA gives us an “early warning” when items start to age. If your conclusion is that you want to decrease Cycle Time, your experiment will most likely include monitoring and acting upon aging items.

In my above example, I focused on slicing work items smaller as an experiment. Another option could be to set a new (lower) goal for Cycle Time, monitor the age of all the items, and start to act as soon as an item gets close to this goal (for example by swarming this item till you manage to close it).

Conclusion

Little’s Law is incredibly helpful for us if we want to design experiments to improve our flow. By knowing the different forms of the equation, we can understand what must change to achieve the result we would like to have.

If we’re designing experiments, we should be aware that both Cycle Time and Throughput are lagging indicators, while Work Item Age is a leading indicator. Moreover, we tend to have some form of control over our WIP, so this is also “simpler” to change and allows for more concrete experiments.

And don’t forget that for a proper experiment, you should observe what is happening in your process and check if you get the results you expected. And if not, you should adjust accordingly.

Resources

If you’d like to use this blog post for an interactive learning session (for example with your team), it’s available as a template on Miro.

I highly recommend reading all posts of Dan Vacanti’s series about Little’s Law, but if every minute counts, at least read:

• https://www.55degrees.se/post/when-an-equation-isn-t-equal

• https://www.55degrees.se/post/littles-law-assumptions

Below are some existing Blog Posts that we wrote around Flow Metrics, the Flaw of Averages, and Controlling WIP:

Special thanks to https://imgflip.com/ for providing me with all the Meme Templates.

In this post, I’ll explain how this came to be, how well it worked, and what we learned from it.

But first, let’s explain shortly how I ended up serving in the military in the first place…

The post uses some military terms that I often hear only in German and that don’t translate well (because they are somewhat military-specific and not used anywhere else I’m aware of). I do my best to describe it, but occasionally I’ll also mention the German word (or abbreviation). Sadly there is no easy translator from “Military German” to “Military English”…

Military Service in Switzerland

In Switzerland, there is a militia system that spans from politics to the army and other areas. While some of it is voluntary, there is a mandatory obligation to join the Swiss Armed Forces or alternative services like civil service for every male Swiss citizen. This means, most units are made up of people that have regular occupations and are called up for military service at regular intervals. After basic training that lasts roughly 6 months and is done at around 20 years of age, you serve in a unit and spend roughly three weeks each year until you’ve served all the time you owe to the state. The number of days you owe depends on the rank - I’m a sergeant and have to serve a total of 400 days (25 days pending at the time of writing).

The National Emergency Operations Centre (NEOC)

The unit I serve in supports the National Emergency Operations Centre (short called “NEOC” or, in German, “NAZ”). The NEOC is a civil organization that is part of the Federal Office for Civil Protection (FOCP).

The main task of the NEOC consists in maintaining an overview of the situation which is relevant for protecting the population.

In the event of incidents involving radioactivity, major chemical accidents, dam ruptures and natural hazards, the NEOC takes on additional tasks. As far as radioactivity is concerned, the centre is authorised to implement immediate measures to protect the population.

Being a civil organization means that there are non-military people who have regular jobs at the NEOC. In case of a large-scale event (issues in nuclear power plants, earthquakes, etc.), they are supported by military staff:

NEOC has a personnel pool at its disposal consisting of well-trained, qualified specialists who are ready for deployment within four to six hours of an incident.

The military staff trains with the civil organization three times a year, to make sure that the processes and tools are well understood, as in case of an emergency, we have to be operational within hours.

Operations Unit / FGG3

The NEOC is split into various “departments” that have different responsibilities. For simplicity, I will refer to them as units.

In German, they are called “Führungsgrundgebiet” or “FGG” (and you might see the term in some pictures below).

I am part of the “Operations” unit (FGG3), which is responsible for the various processes we have, preparing warnings and alarms, and triaging incoming messages to the NEOC.

If this sounds interesting, you can apply to do your service at NEOC, you can find more information about the process to apply online (only available in German).

In case of an emergency, all units will have representatives (civil and military) in a main operations room, while the bulk of the military personnel will work from a “back office” (BO).

In operations, we have various “stations” as we have to create different “products” (documents for supporting the decision-making), send out alarms to warn the general public, and make sure we distribute information on all relevant channels so other units and affected authorities (like police, local government representatives, etc.) are informed.

In the back office, we get tasks from the unit leader after they talk through them with the other units. While we know what kind of warnings eventually have to be sent out (it’s a pretty well-defined process in most cases), various factors influence the content (magnitude of an event, weather, etc.). In the back office, the work is done by a subset of the unit, as an event might take several days to weeks, and the work is done in shifts.

Learnings from our Service

In September 2024, we had a week of service where we did a two-day training exercise. While in general, it went well, there were a few points that were raised during our review and feedback session.

There was some confusion about the progress respectively the state of some of the tasks. If we send out warnings or alarms to the public, they must go through a more rigid verification process, where two other people have to check and confirm that everything is correct (you don’t want to set off alarms or evacuate towns for no reason…). If this takes time, it’s very easy to forget about those tasks:

• Is it being reviewed right now? Or was it already?

• Has it come back for some adjustments?

• Have we sent it out yet?

When certain decisions are taken, the inflow of tasks can be high, and often many things are needed (it goes without saying that all of it should be done as fast as possible). It has happened that some tasks simply were “lost”. Sometimes tasks come in via phone, via digital systems, or via someone just walking from the main operations room to the back office. Combine that with shift changes and you can easily “lose” a task.

Some processes require a confirmation from another party before they are “done”. It’s easy to think that the work is completed with the first part, and we might never get the confirmation. And while everything is documented in a wiki, in stressful situations you probably will forget that you need a confirmation and simply move on.

Transparency

While in my professional life, I mainly deal with teams that work in software development, I’ve come across such issues before: We were missing transparency. Transparency about what is ongoing, what is waiting to be picked up, and what is already done. I assumed that with more transparency, we’d be able to inspect and make sure:

• We know each task we’re currently working on (and who is working on it)

• We know where in the process each task is

• We know the dedicated steps for each task

And with this, we could also take better action on what to focus on. Instead of “just executing”, it should allow us to self-organize the work in the back office. Instead of micromanaging, we collect what needs to be done, and let people work on it how it best fits them.

Kanban Board

The proposal was made to create a Kanban Board where we could track our tasks through a defined workflow. Each card on the board would represent a task and could have details for this specific task written on it. That should help guide the people who take up the task and make sure we know what to do and where we are in the process.

Kanban Definition

Before I go on and share how this was put into practice, let me quickly share what I mean when I talk about Kanban. I’m using the definition as described in the Kanban Guide:

Kanban is a strategy for optimizing the flow of value through a process that uses a visual, pull-based system.

So we want to optimize the flow of value, where flow means “the speed and smoothness of the delivery”.

Good flow feels like a well-oiled machine. The Kanban system members enjoy a sense of calm and get more work done with minimal stress. - Definitions for Kanban Guide

To do this, we planned on introducing a “Kanban Board”. This is a visual representation of a workflow. As the goal is transparency, it should be accessible to the Kanban system members at all times.

Preparation…or lack thereof

Our next service was planned for November 2024, two months after we concluded we wanted to give the Kanban board a try. This specific service was special, as we were part of a big exercise (Gesamtnotfallübung 2024) that happens only every two years and where national and international partners take part (for example the national grid operator Swissgrid, the nuclear power plant in Gösgen, the Swiss federal railways, the Swiss Federal Nuclear Safety Inspectorate, and more).

In light of this, we could not afford to take time and train everyone for one or two days on the new approach. When most of the staff would arrive, the exercise would be in full swing already (it’s part of the exercise that the military staff is called upon during the exercise, as it would happen in a real event). At the same time, there is no better way to try this out, it won’t get any closer to a real event than this.

Two of us were invited a day before the start of the exercise to prepare the basics, but we had to keep it simple. We decided to go with a minimal approach and did the following:

• Design a board that has all the relevant information needed for people to see who is around and what’s going on

• Design a workflow that should fit our purpose but is not overly complex (we had no time to elaborate on explicit policies or complicated WIP control strategies)

• Prepare a template for the Kanban Cards and some pre-filled Cards that we knew we’d need (as the scenario is quite standardized this was possible) so we have a starting point but also have the flexibility to capture things we could not yet think about

On the top right, you can see our shift plan and below the names of the individual people, split by who is currently “active” and who is “inactive”. We created transparency about two specific roles as the “Chief Back Office” is the point of contact and it has to be known whom to talk to, as well as another role (“ASNAZ”) that happens to be in a dedicated room (so it’s good to know who is there as you can’t see them).

We also highlighted important dates (when is the next “situation report” and when do our products have to be updated) which should inform everyone working in the back office.

In the middle part, you can see the workflow, which looks like this “To Do” → “Waiting for Information” → “Processing” → “Product Update & Storage” → “Done”

At the bottom, you can see the Kanban Cards ordered by categories that made sense for the event, as well as some “blank” cards for anything we did not foresee.

We assumed that this was “clear enough” that we could get started without much introduction, while still providing enough value. We were also very much aware that this was far from perfect, but it was not worth putting more effort into - instead, we should inspect and adapt in the coming days.

Obeya

You might have noticed that the above is not showing “just” a Kanban board, but has more information visualized. We implicitly applied ideas from “Obeya”:

An Obeya is a physical or digital workspace where strategy meets execution. Typical about Obeya is that it guides both behavior as well as a work environment design. It creates a so called “single source of truth”. - Obeya Association

In short, the idea is to visualize everything that is needed for the people that work with Obeya so they can make decisions as independently as possible. You can find more information about Obey on the homepage of the Obey Association.

For the rest of the post, whenever I refer to Kanban, be aware that implicitly I also mean the Obeya concept.

Experience During The Exercise

When I arrived at our back office, a few people were already there, but we were not yet operational. Here the board helped a first time - we could tag who is already around (independent of which shift they are in) as well as look at the predefined cards. By going through them, we could in a more structured way try to understand what has already happened and what we should start to take over (the civil employees might have done some of the work already, and the cards helped us to “transfer” the knowledge).

During the first day, the acting “Chief Back Office” (CBO) started to familiarize themselves more with what works (and what doesn’t work). For example, it was not clear:

• Whether everyone should update the board or it’s just the CBO that does this.

• For what we needed a card and what we can simply do.

• When/if a card should move backward.

But overall, the feedback from the group was positive, the board and process seemed to help. And often you could find someone studying the board.

We also had some early learnings and made some adjustments:

• We very quickly ran out of space in the “Done” column, but also wanted to keep the cards (as they are a great artifact for reflection later on). Also, we wanted to see the done items we recently closed (that is valuable information as well), so we simply added an envelope and stored the done items once we ran out of space.

• Some people might not be available (due to a leave of absence, having lunch, etc.). We indicated this by turning their names upside down.

• We also have three specific products, which have to be updated for the situation reports. Indicating who is taking care of them on the board helped to make things more clear.

During day two of the exercise, the board looked like this:

As you can see, a lot more stuff started to happen, and the board proved to be a great radiator of information. It helped the CBO keep an overview, and also make sure we’re following up on our tasks. One example was that we were tasked to organize a briefing for an external group. However, right after that, the situation escalated and lots of more important things came up. It took us till the next day to finally follow up on this task. Without having it on the board, we most likely would not have remembered to eventually follow up on this.

Learnings after the Exercise

After two days of exercise (which went rather well), the feedback on the approach with the Kanban board was rather positive. It helped “track things to closure” organize the work, and in general keep a good overview.

Having the board also uncovered some dysfunctions - mainly outsiders “walking in and walking up” to people in our back office and giving them tasks. This became visible because it was not clear whether cards should be added, whether should we work on this now, etc. This has triggered a discussion about who is making tasks available, who should be “accepting” them etc. which is something we will have to clarify going forward.

While this is not “Kanban by the Guide”, it was very useful. Going for a simplistic approach made sense, given the constraints we had: No real preparation time, no time to train anyone, and also it has to be clear that this way of working might be new to some.

Our unit is diverse, not everyone works in Software Development where “Kanban” might be a common term and easily understood. Someone even referred to it as “Kaboom” instead of “Kanban”. This sparked the inspiration to name the whole process (and the title of this blog post): The “Kanban Back Office Operating Model”, or “KABOOM” in short.

Having predefined cards ready for events that have a defined process is extremely helpful, as when we are there for an emergency, they are a great guide. However, we noticed that we can put more guidance on the card itself (think of them like “predefined subtasks” that we know we have to do for specific actions). Also, we decided to simplify the workflow, as it turned out that two of the stages were not providing value, and instead capturing some of the steps needed (if they are needed for a given task) on the card itself, reducing the workflow to “To Do” → “Processing” → “Done”.

On the cards, we also introduced a step in which we capture “lessons learned” before we set anything to done. Having this happening “in the moment” is a great practice, as often we will forget as time goes on, even if we reflect just a day or two later. But having the cards and some space to write things down as we just worked on it will make sure we’re capturing as much as we can.

The adjusted cards are now stored in our KABOOM envelope. And while we hope we never need it, if it ever comes to it, we simply can grab it and are set up in a matter of minutes.

Another interesting aspect was the “Chief Back Office” and the active shift might use it in different ways. Some might prefer to have more “fixed stations”, while others might use a more flexible approach. This could also depend on the current situation. The great thing is, that the KABOOM will give you flexibility so people can focus on the work and decide how they want to work. Some might call that agility, but I’d say just give people the information they need and let them focus on the work to be done.

What’s Next

Our next service is in March 2025 (and hopefully we don’t have to meet earlier…). The plan is to further refine the KABOOM and also take some time to elaborate on the different elements as there is no big exercise planned.

There is also the question of whether the board should be physical or if a digital version could be more useful. I’m a fan of a physical board, but there are some advantages of a digital one (for example the team in the main operation room could see it) and we might investigate the possibility of it.

Last but not least, being a nerd for numbers, I’m curious if there would be any value in capturing the flow metrics. I’m undecided on this, both because I would not know how to best capture it and in general if it would provide value, but it might be worth an experiment.

In this post, I'd like to share a different perspective, show how teams can profit from transparency around metrics, and what you need to get started.

What's usually being compared?

The metric I've encountered most in such a discussion was Velocity, defined as Story Points closed per unit of time (for example per Sprint), often in the context of "How much can we get done"?

When the debate about comparing teams came up, it was usually to answer those questions:

• Why is this team performing better (meaning more Story Points are getting closed) than others?

• Why is this team planning better (meaning the team is "closer" to completing the planned amount of work) than other teams?

Other metrics that I've seen being used to compare were:

• Number of defects open/found

• "Code Quality" measures like test coverage, code smells, etc.

Often these numbers needed to be reported somewhere, accompanied by a semaphore indicating whether we are "green", "amber", or "red" against some arbitrary target (80% code coverage, anyone?).

Why is this a problem?

Whenever someone would suggest comparing the above metrics across teams, my knee-jerk reaction would be to block and explain why this is a bad idea:

1. The context of my team is different, thus it does not make sense to compare this.

2. Whoever sees this metric doesn't know that the context is different. They will wonder why we are "worse" than other teams, and start to punish the team in some form.

3. The team is smart and knows that, so they will start to game the metric, making it useless not only for people asking for it but also for the team themselves.

Now those problems are real, and I've seen this happening. So why would I suggest to compare teams and metrics anyway?

It's all about the Targets...

Let's start with the last point above, which is the fact that teams will game the metric. You might have heard of Goodhart's Law:

When a measure becomes a target, it ceases to be a good measure

If arbitrary targets are set across the company, it usually doesn't take long and teams start to figure out a way to reach that target.

"What if we exclude these parts from the test coverage, they are legacy and we can't test them, so it's not making sense to include them..."

While it probably comes with a good intention, such targets undermine continuous improvement efforts. Instead of having transparency about where teams stand, you can't trust the data anymore. So even for the team itself, assessing how they are doing might not be possible.

"Hey, this bug actually somewhat is like a feature request in a way, so let's track this differently...".

What I've seen work is to look at trends over time instead of "hard numbers". This brings several benefits:

• It takes the context of the team into account. No matter if you right now have 200 or 3 open defects, 85% or 29% code coverage, you can track if you're improving over time.

• If you look at it over time, you can better deal with the natural variability. Your defect count might spike once for some reason, but over time you should be able to keep it stable or reduce it.

These "relative improvement targets" should then keep the context of the team into account. For example, if you use a tool to track "code smells" (like SonarCloud), instead of saying you should have 0 smells (or less than 1000, or any other random number someone thinks of...), look at how you can prevent new smells from entering your code base, and continuously work to reduce the current issues. Sometimes you might still introduce a smell, and that's an opportunity to improve your process.

This will have a more sustainable impact than forcing to get to 0, as it will lead to rules being disabled (no rule, no issue...), or code being excluded from the analysis. Some might say that teams can still game those numbers. I agree with that, but in my experience, it's a lot easier to explain that our goal is to eventually be at 0 code smells, but for now, we're targeting to not introduce any new ones and clean up existing ones as we go, than simply saying "make it to 0 or else".

Now if you compare various teams, and see that one team manages to get down quicker, you might ask what they are doing differently (maybe they simply prioritize this work, or they have better tooling...). If you see a team where the issues are continuously rising, you should probably also ask what's going on with this team, and how this team can be supported.

...And The Metrics

Next to the targets, you should also make sure that the metric itself makes sense. As mentioned, I've often seen that estimates respectively plans based on estimates are compared. Estimates are guesses by nature, they are not meant to be accurate. If you start comparing teams based on their accuracy, what most likely will happen is that estimates are being blown up. Instead of a "5", this now becomes an "8" or "13", just to be safe.

If you can play with your metrics like this "at will", I would argue that it's not a good metric, and you should not track it. While I'm not a fan of using estimations like Story Points at all, you might still find them useful as a tool for discussion. Just don't use it for anything else.

Instead, use something comparable across teams. Code coverage and the number of defects work better for this because you can clearly define what it means. Instead of estimates, I would propose to use Flow Metrics, as they will give you information about the flow of each team that you can compare with each other.

For example, we can look at the 70th percentile of the Cycle Time over the last 3 months. If you see a team that has considerably lower Cycle Times, that's an opportunity to learn. How are they breaking down work? How do they collaborate to get to done as fast as possible?

Try to refrain from metrics that are not comparable and highly dependent on the team like estimations, and instead look at more "neutral" data. That doesn't mean the numbers will not be different based on the context, but it will allow you to be curious about what they are doing differently, as it's not just "subjective" to the team.

Context Matters - But is not an Excuse for not Improving

Yes, teams have different skill levels and work on different technology stacks. The culture differs from company to company. However, why would this stop you from looking at how others are doing, and thinking about what they are doing differently if they seem to perform better?

Why would we not want to learn what this other team in our company is doing differently? Maybe it doesn't apply to us, but maybe it does.

Instead of "protecting" the team, we should actively look for ways to improve. It's easy to say "Oh that team has more senior developers, that's why they have better numbers". Is it the seniority, or are they just doing something different that you could also try as an experiment? Are you investigating this? Are you having discussions on how the seniority in your team could be increased? Or are you just looking for excuses?

Every team, every product, every context is different. But that is no excuse for not learning from each other.

For example, I've seen a team that had a much higher Cycle Time than another one in the same company. When checking what was different, it turned out that they were relying on a different team that did the verification for them. So the items were "stuck" for quite a while until the other team picked them up.

Once we learned that, there were two options:

• Saying "Oh yeah, of course, you have lower cycle times if you don't depend on this team. But that won't work for us" and not doing anything.

• Taking action and find ways on how to improve the situation.

I guess it's needless to say which one should be the preferred option.

Prerequisites

As mentioned at the beginning of this post, I used to be highly "protective" when it came to metrics, especially making them transparent or even actively comparing them with other teams.

Nowadays, I can see many benefits of comparing the metrics of teams together. My aim with any team I work with is not to be mediocre. We aim that we become the best at what we do. This doesn't happen by hiding numbers, but by being brutally honest. You may not like what you see, but it's the first step to improving anything.

"The truth will set you free, but first it will make you miserable" - Attributed to James A. Garfield

You also should look at what you are currently measuring. Are those metrics useful, or should you measure something else? Data can be incredibly useful, and I urge you to check out Flow Metrics if you haven't yet, as they are powerful, actionable, and very easy to measure.

You should educate your team, as well as your organization. In the past, I accepted that we had to fulfill random targets, and was ok with finding any way to get to this number. In retrospect, I was just not doing a good job and making my life easy. Thinking that "management anyway doesn't get it" is an excuse that will lead to no improvements. In reality, people might simply not know better. If you approach them and explain why for example looking at the improvement trend over time might make more sense than a hard number, you might learn that they are happy to learn from you.

Accepting "faking" metrics is not only unethical, but you also rob your company of the chance to improve. If something is not ideal, you should not be hiding it. You should make it transparent and ask for support to improve it.

These prerequisites are especially important if you are in some kind of Scrum Master/Agile Coaching role. This is your job. Go do something to improve the environment to create transparency. Without it, you won't ever establish any kind of continuous improvement.

You may not change your complete organization at once, but start with what you can control. Create a dashboard, or even better, an Obeya with the data that is relevant for your team. Invite others to look at your metrics, and see what happens. Actively ask how you are doing compared to other teams, and be open for input.

Conclusion

Using metrics can be incredibly helpful to improve within a team itself. If you collect useful metrics, you can base your retrospectives on data. However, if you do it across many teams, you can supercharge your continuous improvement efforts and cross-pollinate knowledge across teams.

If one team seems to be "better" (whatever that means in your context), it does make a lot of sense to be curious and ask what they might be doing differently, instead of just assuming "their context is different, we don't want to hear anything about it".

Work with your teams and organization to:

• Move from fixed targets towards continuous improvements over time, by looking at trends and making sure you improve.

• Move from metrics that are subjective to the context of a team towards metrics that can be compared (keeping the context in mind), for example, Flow Metrics.

• Move from secrecy around metrics of the team, towards transparency to enable inspection and adaptation, for example, by creating an Obeya for your team.

If you are interested in learning more about Flow Metrics, Obeya, or Data-Driven Decision Making, check out our training or reach out for a call.

In this article, I’d like to introduce you to a team I was fortunate enough to be part of. It’s a pretty common team, one that probably most of us have met or been part of during our time working in teams. It is probably a bit too large in team size and has team members with a variety of skills, with the aim of delivering software to customers. And, like other teams in the realm of software development, something was not quite right with this team — you could say that this team was stuck.

I was brought in as an external consultant to work with the team, as usual with my engagement in the form of ‘do something with the team and improve it.’

In this case study, I’d like to invite you on the journey we took, how we were able to get unstuck, and even more importantly, become a team of individuals that creates value — with some support from data that we always had but never thought of using.

Question to myself: Any regrets that I delayed writing the article?

It’s actually the opposite — I’m quite happy about it, because it was only when I got my hands on Jim Benson’s book “The Collaboration Equation” that I truly understood what was happening in this specific team I will write about.

The book, which is already my ‘must-read for 2024 recommendation,’ gave me the understanding and also the right words to describe what this team went through and how they moved from their learned helplessness to a state in which “individuals in teams create value.”

Learned Helplessness

Let’s start with the definition of Learned Helplessness — Jim Benson describes it as:

Learned Helplessness is the tendency not to act at all because you increasingly expect the system to thwart you and then penalize you for even trying. Logically, it is safer to not act (to give up your agency). It is easy to blame the victim and tell people to go that extra mile, but learned helplessness isn’t simply giving up. Learned helplessness is the result of realistically assessing no-win situations and realizing that action will yield no or negative results. In an anti-collaborative environment, acting with confidence is dangerous, which means that professionalism is dangerous — Jim Benson/The Collaboration Equation

Everyone working in this team would have noticed symptoms that were visible in this team — there we plenty of them.

If I had to pick three they would be:

• not able to control their own working environment

• a very limited flow of information within the team and outside the team

• a self-fulfilling team identity of “that’s just how we are”

The last point is the most interesting to me, as it is also something often observed in individuals. After some time, we tend to believe a story others tell about us or that we tell ourselves. And given that the collaboration equation is ‘Individuals in teams create value,’ maybe it’s not so surprising after all.

What we usually do vs. what we should do instead

So what was it?

Did we start by changing the people?

Were these individuals who simply didn’t care?

I guess the usual tendency in most companies nowadays is to focus on the performance of individuals, conduct awkward yearly reviews, and just hope things will improve. And if not, they start looking again to find who could be accountable for the situation, change people, change managers, keep everyone seemingly busy, and yet generate no real change at all. It’s like being in a constant loop.

From my point of view, there is a better question we could ask instead:

Do these professionals work in an environment/system that allows them to be good professionals?”

As Jim Benson puts it:

No incentives, no negative reinforcement, no threats to their livelihood, can get people to perform well in an environment which does not give them the freedom or even the ability to perform well — Jim Benson/The Collaboration Equation

So instead of working on the people, let’s have a look into the system they worked in.

Collaborating when it was not avoidable

The team was working within a system that enforced the use of Scrum. To avoid any conversation that might completely miss the mark, I’d describe it more accurately as a “push” system.

Every two weeks, new work was pushed onto the team’s workflow.

While there was initial feedback from the team about this situation, it eventually subsided over time, leading to a state of learned helplessness within the team.

Now comes the most crucial part of it:

This situation dragged on for more than four years. During this time, every two weeks, new work was pushed into the system, regardless of whether the previous tasks were completed or not.

This often resulted in a sprint backlog of over 70–80 tasks, with a team struggling to survive week by week. The situation reached a point where even frustration wasn’t openly expressed anymore, as it became accepted as “just how things are.”

Team members simply picked up another ticket, worked on it, and moved to the next one, collaborating when it was not avoidable.

I’m pretty sure that if this team would have looked at the 5 dysfunctions of a team, we would have scored quite high in all of them.

And every attempt to help the team from the outside, came from the perspective of “fix them” and “make them understand” and never looked at the system the team was working in and if the system made it possible for good professionals to actually be good professionals.

Fix the system, not the people

So what happened?

Hopefully not surprising to a lot of my readers, it wasn’t a big transformation that helped this team move into the right direction.

What might be surprising is the fact that it all started with a simple question:

“What if we would just stop working as we do now?”

And, although it might have seemed like a fairly standard question from the outside, the reactions on the faces of those we were looking at on the screens told a different story. It appeared that in four years, no one had thought to ask this team what their ideal working environment would look like.

This was the beginning of what I call our “culture bubble” story.

Check Evolve2B for more information on Culture Bubbles

How the right environment looked for this team

I have to admit that I myself didn’t expect this to work as well as it did.

But more suprisingly, I was shocked on how quickly these professionals were able to be in charge of their professionalism again.

The first step was an obvious one:

We changed from a push system into a pull system. Work was not longer being pushed into the system every 2 weeks but instead was waiting in a backlog to be pulled by the team. We still kept our iterations of 2 weeks for 2 reasons:

• we wanted to keep a dedicated time we would get together and discuss priorities, learnings and also look back at what we achieved

• the second reason is that the system around kept on working in iterations of 2 weeks so we decided to build adapters and this way were able to reduce noise regarding the changes we have made and at the same time, were aligned with the teams outside the team

A next step was that we wanted to make sure that the way our work flows becomes visible — to us, but also to the people relying on our work.

So the look of our working board changed — we have added new columns that we hoped would make the progress of our work more visible:

We went from

to

As another point of discussion, we talked about Flow Metrics and decided that the data they provided could be a valuable indicator of our work.

This data would provide us with information, allowing us to act with confidence. We agreed to give this approach a try and have another conversation about it in six weeks, at which point we would decide if this was the right path.

I’d love to tell you more about what we did — about all the amazing workshops we conducted with the team, about the fruitful one-on-one sessions, and how we unlocked deep-seated personal issues.

But the truth is, we didn’t do anything else. It wasn’t that these professionals were incapable of working together, or even worse, unwilling.

Rather, the image that formed in my mind was that of individuals at work, chained to a wall. In this scenario, all that was necessary was to help them realize that they could break free from the chains themselves.

I already told you at the beginning of this section how quickly it went.

Well…how quickly did something change?

Professionals who are in charge of their professionalism

It took about two to three days for the new working system to click in.

Cleaning up the backlog was obviously necessary, and this wasn’t a task performed by a single person but rather a team effort.

Getting accustomed to accurately reflecting the status of our work on the new board was probably the hardest part for everyone on the team.

But once we overcame that, collaboration and work began to flow smoothly.

So, what did we see happening next?

Fruitful conversations about work in progress became a regular part of our daily huddles.

We shifted from dull daily meetings to ones where we:

• Actively engaged in conversations about how we could collaborate to move aged items to the right side of the board (action on data)

• Team members sought support from others

• People decided to pair up for specific complex analyses or tasks

• Challenged priorities using the information that was visible and available to us

Eliminating value-negative tasks

• While sitting in meetings, we openly challenged their value, quickly agreeing to try alternatives. Mostly it was along the lines of, “Let’s see if we miss anything if we don’t have this meeting.”

• The board introduced new ways to quickly and effortlessly find information.

• We implemented quick fixes swiftly. For example, “Could we visualize information for the person who picks this up next right on the first screen? It would then be easier for them to decide what to do next.” This meant, for instance, letting the team member who would take over QA know in advance which environment was needed for testing:

The last part was an interesting one, as it happened naturally.

No Retrospective was performed for these changes. These were all things that simply happened or that were brought up during a daily huddle.

Quickly identifying bottlenecks and acting on them

Thanks to our new way of visualizing our work, we were quickly able to identify bottlenecks within the system we were working in. To my great surprise, even though most of these bottlenecks were beyond our direct control, the team realized that there was something they could do to alleviate these choke points to some extent, and they swiftly took action on it.

There was also an instance where the team, without any external interference, started discussions about having smaller work items and how it would benefit their feedback loop. (The 85th percentile cycle time proved quite useful in determining a good starting point for this.)

It’s important to note that we never started with a Work In Progress (WIP) limit. However, due to the collaboration and the pull-system in place, it never became a real issue. If a problem did arise, it was quickly addressed in the daily huddle, actions were taken, and we moved forward.

Increase in engagement

The points mentioned above already suggest an increase in engagement. But one aspect that really stands out to me is the atmosphere you experience when you enter a room with these individuals. There’s a palpable difference in their demeanor and in the way they work.

We see systems that engage people professionally, leaving them energized at the end of the day and not tired and spent. — Jim Benson/The Collaboration Equation

Give me the data

Although we rarely discussed the data itself, simply accepting it as information to enhance our team decision-making and our collaboration, we were still surprised by the results of the first data set.

This is what we got:

The graphic above shows what we were capable of achieving in the three months prior to the changes: a throughput of 140 items, with each taking 37 days or less to complete (as indicated by the 85th percentile, highlighted in light green).

When we took another measure after the first 3 months, and got the following:

With the same scope of three months, we now achieved a throughput of 186 items and reduced the cycle time from 37 days or less to 17 days or less (again, refer to the light green line for the 85th percentile).

Also, take note of the changes in the Y-graph.

Although we still had outliers, as unexpected things happened, we took action or were consciously aware of them. The more our items aged and approached our 85th percentile line, the more we discussed them in our huddle.

There is one aspect that is difficult to measure, and perhaps shouldn’t be, but it’s one that keeps this team continuously improving in their collaboration: the increase in the number of professionals who were empowered to work effectively.

A key learning for us was that having data helped shape our conversations within the team, leading to more informed and effective decisions.

What came after?

Understandably, people around this team became curious about how they managed to break free from their previous state and are now trying to adapt their systems to achieve something similar. This is the beauty of a “culture bubble”— extensive promotion isn’t even necessary.

People from the outside quickly notice the change and start asking questions.

Using data also opens up new opportunities to answer questions like, “When will it be done?” (Thank you, Monte Carlo simulations).

This hopefully serves as another example of how visualizing our work, and planning future tasks, aids in collaborating with others and making informed decisions.

Another interesting aspect is delving into the concepts of Obeya, as described in Benson’s book (Did I mention that it’s an absolute must-read?).

To me, the most significant takeaway remains the fact that no matter what we do, working within a system that allows individuals to excel in what they do best is crucial for awakening the professional within them.

Individuals in Teams create value- — Jim Benson/The Collaboration Equation

Do you want to learn more about Flow Metrics and how to shift the dynamics in your team? Then join one of our upcoming micro-sessions.

Check them out here: https://letpeoplework.eventbrite.com/

In this post, we’ll explain what a Monte Carlo Simulation is and why this might interest you, and we'll walk you through the different steps of how it works.

This post was first published on Medium and republished in an updated version.

What is a Monte Carlo Simulation?

Wikipedia describes the Monte Carlo Method as follows:

Monte Carlo methods, or Monte Carlo experiments, are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. The underlying concept is to use randomness to solve problems that might be deterministic in principle — Wikipedia

To forecast, we try to “simulate” the past and apply it to the future. We run many of those simulations and see what is “most likely” to happen. It’s a “probabilistic forecast”, meaning we don’t get a single result, but many options that are associated with a probability.

How Many & When

When applied to forecasting in software development, we can use the Monte Carlo Simulation to answer two questions:

• How many items do we manage to close till a specific target date?

• When will a specific amount of items be done?

Having answers to those questions is very powerful when we’re planning, and Monte Carlo Simulations allow us to answer this easily without estimations in just a few seconds. More information can be found in Daniel Vacanti’s books “Actionable Agile Metrics for Predictability” and “When Will It Be Done?”.

Monte Carlo Algorithm

The goal of this post is to make it more clear on how a Monte Carlo Simulation works. I’ll first explain the algorithm on a high level and then go more into the details. Some terms, like Throughput, might not be clear, please bear along with me as I’ll elaborate more on them during the post.

A simplified version of the algorithm to make forecasts with Monte Carlo Simulations looks like this:

1. Create a run chart of your Throughput

2. Randomly select values on this chart and sum up the Throughput. This is the simulated Throughput for a single day. Do this either for the number of days you want to simulate (“How Many”) or till you have 0 items remaining (“When”).

3. Write down the results (either the sum of the Throughput or the amount of days).

4. Execute steps two and three many times (like 10'000+ times)

5. Plot the results on a histogram. Identify the “risk profiles” that you are interested in, for example, what you got in 85% of the simulations.

Before we start with step 1, the Throughput, let me elaborate on our sample data.

Sample Data

In our opinion, it will help you follow along a bit more actively. To make it more engaging, we’ve created a spreadsheet that you can use to follow along. You can find a copy of it on Google Docs or on OneDrive if you prefer MS Excel. Ideally, you have it open on the side while reading. I‘ve marked the parts relevant to the spreadsheet with this icon:

In our example, we’ll be using the Throughput of a team from December 2023 and we’ll forecast:

• How many items will be done in the next 30 days?

• When will 28 items be done?

Throughput

The only thing we need to run a Monte Carlo Simulation is to have data on the team’s Throughput.

Throughput: The number of work items finished per unit of time. Note the measurement of throughput is the exact count of work items. — The Kanban Guide

The unit of time can be anything you want it to be: days, weeks, Sprints. For the sake of this post, we’ll be working with days.

Note: If you would use another unit of time, say weeks, the Monte Carlo Simulation would also work with this unit. We would forecast how many items we manage in a certain amount of weeks (instead of days) and how many weeks it takes to complete several items.

To calculate the Throughput, we simply need to know on which date your items were closed. If you are using a digital tool, the chance is very high that you already have that available. If you are using a physical board, just note down the date when an item was closed.

What about the Other Flow Metrics?

You might be thinking, wait a minute, what about the other measures of flow? Are WIP, Cycle Time, and Work Item Age completely irrelevant for Monte Carlo Simulations?

The short answer is yes. You don’t need them to run a Monte Carlo Simulation.

The longer explanation is that, while you don’t need them for the Monte Carlo Simulation, they still are an indicator of how “good” your forecast will be. Controlling and taking action on those measures, helps us get more predictable Throughput. And the more predictable our Throughput is, the “better” we can forecast.

Creating a Throughput Run Chart

Our Monte Carlo Simulation needs the Throughput ordered by days.

In our spreadsheet, you can check out the “Throughput” sheet. In column A, we see the dates when items were closed.

You can see that some dates appear multiple times, this means several items were closed on that day.

We now create a run chart with this data. As mentioned above, we use the data from all of December, so we create a table that shows how many items were closed each day in December:

We can also show this in a chart:

This will be the base for our simulations. You can see there were days where the team managed to close 4 items, while on other days nothing was closed.

What about Holidays and Weekends?

The attentive reader has probably noticed: We include weekends (like the 2nd and 3rd of December) and the whole holiday period around the 25th of December. We use the Throughput per unit of time, independent of holidays, weekends, and people being sick or on vacation.

When using Monte Carlo Simulations, we can ignore such things that often make “capacity planning” tricky.

Simulate a Future that looks like the Past

What the simulation helps us to do, is that we use the “past” in the form of our historical Throughput, to simulate the future. People will have weekends in the future (I hope), and they will also be occasionally sick. This is a strength of Monte Carlo Simulations over traditional capacity planning, as such events are factored in.

Now you should put some thought into which Throughput period you select as a base for your simulation. The month of December was randomly chosen, and you might not want to use two weeks over Christmas when everyone was off.

As a rule of thumb: use a “history” that you believe will look similar to the future you’re trying to forecast. If you’re team just formed a month ago, don’t try to use a 3-month history.

Also, if you wonder if you could use more special sampling models (for example use Mondays to simulate Mondays), we recommend this in-depth post by Prateek Singh.

Now that we have our history in the form of an ordered Throughput run chart, we’re ready to run our Monte Carlo Simulation!

Monte Carlo — How Many

If we do a “How Many” simulation, we have a specific “target date”. We want to forecast how many items we can finish in this period.

If you are following along in our spreadsheet, move to the “How Many” sheet. You can read along and then run a couple of runs of the simulation yourself — follow the instructions in the sheet.

Let’s assume we want to forecast what we manage in the next 30 days.

For a single run of our Monte Carlo Forecast, we will simulate those 30 days. For this, we need 30 random values (one for each day) between 1 and 31 (because our history from December has 31 days).

The random number we get is used to “look up” the Throughput value from this day in our history. For example, if we get a 20, we’ll use the Throughput value from day 20 in our history (which would be 1 in our case):

If you do this for all 30 days, you’ll end up with a number. This is our simulated throughput through 30 days.

We encourage you to try it out, play with different values, and check the results.

We should now take note of the result, where we count how many simulations had the specific result. For my example, we would now increase the count for 27 by 1.

As we want probability, we should run this more times. Often about 10'000 times is used to get meaningful results. Feel free to run some runs on your own, for example, 10, and create the result tables for yourself.

After you run this several times, you’ll end up with a table that contains the results:

Tables are not ideal to visualize this, so you can also plot this in a chart:

You can see that the very high and very low numbers happened less frequently. This is because you are very “unlucky” if you hit often days that have 0 or a very high Throughput (in our example 4).

To make this useful for our forecast, we can see how many runs managed to close “x items or more”. In this example, we have 100'000 runs. This means, 1% of the runs are at 1'000. So we sum up the runs, ordered by most items closed. When we get to 1'000, we know that “There is a 1% chance to close x items or more”. In our example below, the items closed at the 1% mark is 47. So in other words, there is a 1% chance that we manage to close 47 items or more in the next 30 days.

While 1% might be a bit too risky for most teams, you can apply the same strategy to look for more meaningful “risk profiles”, like 50%, 70%, 85%, or 95%.

If you are following along in the spreadsheet, identify those risks and how many items we manage to close.

And that’s it, that’s how you can run a “how many” forecast using Monte Carlo Simulations.

Monte Carlo — When

While we had a “target date” or a specific amount of days when doing the “How Many” simulation, we have a different situation for the “When” simulation. This simulation answers the question: How long will it take to close x items?

If you are following along in our spreadsheet, move to the “When” sheet. You can read along and then run a couple of runs of the simulation yourself — follow the instructions in the sheet.

Let’s assume we want to forecast when we manage to close 28 items.

Before, for a single run of our Monte Carlo Forecast, we simulated 30 days. As we don’t know the days, our simulation will now run until we close 28 items or more. We again need a random value between 1 and 31 (as our history is the same as before). We look up the Throughput value from this day and subtract the number from the remaining items.

We do this until we are at 0 (or below). That “day” is the result of this simulation:

In the above screenshot, you can see that it took us 16 days to “close” 28 items.

We should again take note of the result. Before we had a specific number of items in a fixed amount of days. Now we want to count how often each day was the one where we got to 0. For my example, we would now increase the count for 16 by 1.

If you follow along, feel free to play around with the numbers again and see the different results you’re getting.

Again, we should run this more often. Then, we’ll create a table with the results just as with the “How Many” simulation:

We can now apply the same logic as before, and identify the different risk profiles. As an example, there is a 52% chance to close 28 items in 30 days or less:

Congratulations, you now ran both a How Many and a When forecast using Monte Carlo Simulation.

How Stable Throughput Affects Monte Carlo Simulations

In the beginning, we mentioned that we only need Throughput for the Monte Carlo Simulation. However, the other flow metrics might be an indicator of how well the Monte Carlo Simulation will work. As you now should have a solid understanding of how it works, this might make more sense.

If we were perfectly predictable, we would have a “stable” Throughput. This means, that every single day (or whatever unit of time you are using), we would have the same value. If that were the case, every single run of the Monte Carlo Simulation would result in the same result.

The original Throughput sums up to 32 items for December. You can go back to the Throughput sheet, and change all entries in Throughput/Day to 1. That adds up to almost the same total, but what happens with the simulations if you run a couple of them? What do you observe?

On the other hand, if you have a long period where there is nothing closed, and then suddenly a large amount, you would have more “extremes” in your simulations. As most of the time you will select days with 0 (as the chance is higher), but for some runs you might get many instances where you pick the “high Throughput days”. This will increase the “spread of the results”.

Again, what if we change how the total Throughput of 32 is distributed over the whole month? Change the Throughput/Day so that the team would have had many days with zero, while there are just a few days where there was a lot of stuff completed.

This means, a focus on flow, making sure your items don’t age too long, your WIP is not too high and you are working in small batches will affect the Monte Carlo Simulation.

There are also more advanced methods to analyze how predictable your Throughput is: The usage of Process Behaviour Charts with so-called XmR charts. However, that would go way beyond the scope of this post. You can find more information about this in Daniel Vacanti’s new book “Actionable Agile Metrics Volume II”:

Tools for Monte Carlo Simulations

Before we wrap up, we wanted to share some tooling for running Monte Carlo Simulations. It’s nice to run it manually, but not very practical. In the end, you want to have it automated to run it frequently, or ideally even automated, so that you can move towards continuous forecasting.

Write your Own

As you went through the algorithm now, we hope that you saw that it’s not complex. You should have a computer to run many simulations, but if you can write code, you should be able to write a script that does the calculation for you.

The advantage of this approach is that you can customize it fully to your needs. It allows you to integrate with your systems, and your pipelines, and take all the special cases you have into account.

Lighthouse

If writing your own is not an option, you might look at Lighthouse. It's a tool we've developed to continuously run probabilistic forecasts.

You can run forecasts for Features with one or many teams involved and also do "manual" forecasts (When and How Many) for each of your teams:

Find out how to use it and get a link to the Demo on our dedicated Lighthouse info page.

MonteCarloCSV

If you won't or can't use Lighthouse, you might look at MonteCarloCSV.

It’s a Python script that works with any CSV file as input. Just export the “closed dates” of your items into a CSV, feed the script with it, and your simulations can be run.

This might be a good option if you don’t want or don’t know how to build your tool. You can also use it as an easy way to try Monte Carlo Simulations out, without investing too much time. If you like it, you can even start building your tool based on the existing code.

Last but not least, the code is open source and you can run it on your system. Your data does not need to leave any of your systems. If you or your company is concerned about where their data ends up, this might be the tool for you.

Actionable Agile

Actionable Agile (AA) is “the tool” for all things flow. It’s very powerful and integrates with Azure DevOps and Jira. It’s a SaaS offering, and you can find more information on their website.

FlowViz

Last but not least, if you are using Azure DevOps or Jira and rely on Power BI to visualize your data, FlowViz might be a great alternative for you. It’s free to use and very powerful. It can run Monte Carlo Simulations as well as display other useful information for analyzing the flow of your teams.

Conclusion

In this post, you went through the steps that are needed to create a forecast using a Monte Carlo Simulation. You created both a “When” as well as a “How Many” forecast. Furthermore you saw that the only thing you need to do this is to have the data on the Throughput of your team. And that’s something you most likely have already!

I hope the post helped you to demystify Monte Carlo forecasts and gave you a good insight into the method itself.

As you now know that you don’t need too much to try it, I’d encourage you to give it a try yourself. If you do, let me know how it went!

Additional Resources

Following are some additional resources that we recommend to check out to learn more:

Monte Carlo Simulations and Forecasting

Actionable Agile Metrics for Predictability: An Introduction

When Will It Be Done?: Lean-Agile Forecasting to Answer Your Customers' Most Important Question

There is a lot of noise in the daily life of companies — especially if you are part of a big corporation. By now, most have undergone at least one transformation and have adopted some form of an agile framework including several tools that come with it — whether useful or not.

Fact is: These tools are being used and if you are one of the lucky ones who works in a proper product-driven company, this article is potentially not as useful.

In this article, I’d like to share examples of how I have seen one particular tool used and how a simple method helped to open discussions that were not possible before. It’s something that I would encourage everyone using the tool to try out for themselves.

The tool I’m talking about is of course:

Story Points

What are Story Points anyway?

According to Atlassian:

Story points are units of measure for expressing an estimate of the overall effort required to fully implement a product backlog item or any other piece of work. Teams assign story points relative to work complexity, the amount of work, and risk or uncertainty. Values are assigned to more effectively break down work into smaller pieces, so they can address uncertainty. Over time, this helps teams understand how much they can achieve in a period of time and builds consensus and commitment to the solution. It may sound counter-intuitive, but this abstraction is helpful because it pushes the team to make tougher decisions around the difficulty of work.

And even though not required by a framework like Scrum, most teams I have come across do use Story Points and some form or Planning Poker to align on the effort that team members think a specific work item will require.

And while this conversation can be useful, in a lot of teams it ends up being fruitless conversations about “Is this 2 or 3 Story Points?”

Other frameworks, like a famous scaling framework, propose the idea of “normalized Story Points” to help predict the story point size for Epics and Features — meaning find a story that requires 1/2 day for coding and 1/2 for testing and call this story your “1 Story Point” work item. Then give every developer or tester in your team 8 Story Points to work on during an iteration and off you go!

The Ugly Mess

When joining companies nowadays and hearing about the organization using Story Points, I have to admit that my alarm bells are going off.

The first thing I do is to check the understanding of what Story Points mean, so I usually ask the open question: “What are Story Points to you?”

What comes next is usually a mixture of all the things stated above and “but we are not taking complexity into account, right?” is being shared every single time in these conversations. Which then usually leads to more proof that the understanding of what is even being done is not aligned within teams and companies.

I usually then try to throw Ron Jeffries into the mix and share that the person who invented Story Points even apologized for it.

I like to say that I may have invented story points, and if I did, I’m sorry now. Let’s explore my current thinking on story points. At least one of us is interested in what I think. — Ron Jeffries, potential inventor of Story Points

But it seems like the arguments and ideas in the article by Jeffries were still not good enough.

Go Where It Hurts

So my next move was usually to see what benefits companies see in using Story Points to estimate OR where they become a challenge — and very quickly you saw an ugly picture of velocities being (mis)-used to predict the future of teams.

Be it for Sprint Planning, the quarterly planning or to predict when an important feature/milestone will be reached.

I have seen a lot:

• Teams tweak their velocity because humans know how to trick a system

• Complex Excel Sheets in which the Average Base Velocity was calculated and then mapped to the availability of team members

• Re-estimating a story during the sprint closing

• Weeks spent on getting the capacity and story point estimates into a tool for an entire quarter

But what almost all teams and organizations had in common was that there was some dissatisfaction on the management level about what teams committed to deliver vs. what they delivered.

This usually led to statements like:

• Teams need to estimate better

• They need to spend more time refining the stories

• The preparation week needs to be extended (for quarterly planning)

• We need more Agile Coaches

• We need more team members

And I could probably go on for even more. At the same time, I was not able to get my message across that these things will not necessarily improve the situation, for different reasons. What I was lacking was a visual system that would give me a seat at the table to discuss what was going wrong.

Scatterplot to the Rescue

Have you tried putting the Story Points on a Scatterplot and mapping Cycle Time to it? - Well, you know….No

This is at least how a conversation between Benjamin Huser-Berta and me must have looked like.

But my answer was honest — I have never even thought about the option of actually reflecting on estimates and mapping them to Cycle Time.

You might wonder, what do you mean by Cycle Time?

Cycle Time: The amount of elapsed time between when a work item started and when a work item finished. - The Kanban Guide

And here is the thing: Using data was not something I was very fond of in the past. I have considered myself as someone who is simply not interested in it and therefore protected myself from ever digging into it — in Coaching we would probably call it “limiting belief”.

But here I was mapping Story Points to Cycle Time and the results I got were: SHOCKING.

You would assume that there is some correlation between story points and time. Not always, as there is variance in complex work. But in general, you would assume that a 13 takes longer than an 8, which takes longer than a 2.

Most of the Scatterplots looked like the ones above, they showcased there was almost no relationship between estimated Story Points and the Cycle time. This was indeed a problem as these organizations used the “normalized story point” approach.

So whenever they did plan their quarter, their firm belief was:

1 Story Point = 1 day of work

Only to find out by the end of the quarter that their delivery rate was again not even close to their desired goal.

As we can see in the 2 Scatterplots above, 2 Story Points could mean a range from a few days of Cycle times up to almost 160 days. Or that a Story estimated with 10 Story Points could take just as much time as a Story with 1,2 or whatever number of Story Points.

This was not a localized problem but something I can see happening for a lot of teams, regardless of the framework they are working as seen in the scatterplot above. A team working with Scrum and wondering why they are not able to finish what they plan to do.

A conversation about how their cycle time (85th percentile) is 20 days or less, allowed us to have a good conversation on why this team is struggling to finish things in a 2-week sprint. A conversation about, what we can learn from the other items (below the red line).

Seat at the Table

And this to me is the real magic and provided me with a valuable lesson for the future: Visualizing something fosters dialogues and conversations about what matters.

Bringing this scatterplot into meetings with management and teams shifted the conversation from:

“They need to estimate better.”

towards

“We might have a systemic issue here.”

With very little effort, we were able to give people visualized information they were able to base their conversations on. It was not information they all individually had in their heads, but it was visible on their screens, in their meeting rooms and finally, a dialogue was made possible.

The visualization helped to understand what was happening in the system and why this department was getting the results they were getting.

We gave them an idea of their reality.

My Recommendation

It’s very tempting in a situation like this to now hire experts and have them work with the team to improve estimates and work on the symptom. Start looking for who is accountable for the situation, change people, change managers, keep everyone busy, and generate a new kind of noise which will probably dilute the issue at hand for a moment.

In a situation like this, my recommendation is and was to start looking into what you already have. Look for patterns that look different, see what you can learn out of them.

Like with the team above — we can see that this team was able to finish most of their work items in 10 days or less. Go and talk to the team — what do they do differently? What can we now do with this information? Is this a concept that would help us with the challenge at hand?

Once you have the conversation, you can bring in different ideas which yes, could include paying closer attention to Flow Metrics and using your Throughput to do Monte Carlo Simulations that will give you a better prediction.

Try It Out Yourself

If you are in an environment in which Story Points are being used to predict the future (Sprint Planning, Quarterly Planning, etc.), take a look at the tools we provide for free.

FlowMetricsCSV

With FlowMetricsCSV you can extract your issues from your work tracking tool and create a Scatterplot with your data.

FlowPulse

With FlowPulse you can automate the process, and instead of extracting the data into CSV, you can directly hook it up with either Jira or Azure DevOps.

If Python is too tricky for you (trust me though, if I was able to do it, you can do it too), you can also do it manually and extract your completed issues from the last quarter into an Excel file-take the Cycle Time and Story Points for your input and create a Scatterplot in Excel.

Using Scatterplots gave and still gives me a lot of answers on why certain things do not work for teams and organizations — i.e. a cycle time of 19 or days or less (85th percentile) for teams that work in a 2-week sprint, says a lot about why a team is not able to finish their tasks within a specific time frame.

As stated above: It gives us a glimpse into reality and opens doors for fruitful conversations.

Use What Makes Sense, But Do Not Lose Sight on Alternatives

Hear me out: I don’t ask anyone to stop doing something that works for anyone. If you are using Story Points (or any other form of estimation) and it works for you and your teams, good job. Keep it going, keep reflecting, and listen to the feedback you get from the inside and the outside.

Also if you read the article and think: What is he even talking about? We do proper product development, we work closely together, work quickly to verify assumptions and then get on with the work that is upcoming and do not even waste our time thinking what happens in 2 weeks, let alone 3 months, I’m happy for you and the work environment you seem to work in.

My reality though is that there is still a big demand for predictability in companies (especially bigger ones) and I’d ask everyone to be curious about other options to predict what can be done and when something can be done.