Why We Forecast in Scrum Instead of Commit

The Challenge

A common misunderstanding leaders and managers have when moving from Waterfall to Scrum is that the team is committing to delivering everything in the Sprint Backlog by the end of the Sprint. They view the Sprint plan as a project plan. The team committed to delivering the scope in the project plan on a specific date, so why wouldn’t they commit to delivering everything in the Sprint Backlog at the end of the Sprint? It's not that simple. It's beyond complicated, it's complex.

Scrum and The Complex Domain

Scrum was created to solve problems in the complex domain through adaptive solutions. The Cynefin Model helps us understand what defines problems in the complex domain.

Problems in the Complex domain are not predictable. In the Complex domain, the relationship between cause and effect is not known upfront and can only be deduced in retrospect. We have unknown unknowns. Said another way, we don’t know what we don’t know. That means no matter how much analysis we do up front, we cannot accurately predict what will happen. This is why spending endless hours doing analysis, creating a project plan and committing to delivering scope by a specific date when developing a complex product is not the right approach. Decades of experience tell us that using Waterfall, an approach that works well to solve Complicated problems, to solve a Complex problem like software development will lead to delays, cost overruns, lower quality, dissatisfied customers and other challenges.

What’s the Scrum Guide Say?

Commitment – Scrum Value

 “The Scrum Team commits to achieving its goals and to supporting each other. Their primary focus is on the work of the Sprint to make the best possible progress toward these goals.” – Scrum Guide

Here, the Scrum Team is pledging its commitment to do their best to achieve their goals. The “best possible progress” is not a guarantee and it’s not a commitment to deliver everything in the Sprint Backlog.

Commitment – Developers

“Developers are the people in the Scrum Team that are committed to creating any aspect of a usable Increment each Sprint.” - Scrum Guide

A usable increment does not guarantee everything in the Sprint Backlog will be done, only that what does get done will be usable. They collaborate and work together as a team to do their best to achieve their goals.

Commitment – Artifacts (Product Backlog, Sprint Backlog, Increment)

“These commitments exist to reinforce empiricism and the Scrum values for the Scrum Team and their stakeholders.” – Scrum Guide

The Commitment for each Scrum Artifact provides a goal for the team to focus on, measure progress against and enhances transparency.

Product Backlog Commitment – The Product Goal

“The Product Goal describes a future state of the product which can serve as a target for the Scrum Team to plan against. The Product Goal is in the Product Backlog. The rest of the Product Backlog emerges to define 'what' will fulfill the Product Goal.” – Scrum Guide

If Product Backlog Items emerge to fulfill the Product Goal, how can the team commit to work that hasn’t been identified or created yet?

Sprint Backlog Commitment – The Sprint Goal

“The Sprint Goal is the single objective for the Sprint. Although the Sprint Goal is a commitment by the Developers, it provides flexibility in terms of the exact work needed to achieve it.” – Scrum Guide

There is a reason the Scrum Guide describes the Sprint Goal as flexible in terms of exact work. It allows room for the team to adapt during the Sprint based on work that emerges. When describing what the team is going to do when creating the Sprint Backlog in Sprint Planning, the Scrum Guide uses the word forecast, not commit.

“Selecting how much can be completed within a Sprint may be challenging. However, the more the Developers know about their past performance, their upcoming capacity, and their Definition of Done, the more confident they will be in their Sprint forecasts.” - Scrum Guide

The wording is important here. A forecast is not a commitment and allows for flexibility.

“If the work turns out to be different than they expected, they collaborate with the Product Owner to negotiate the scope of the Sprint Backlog within the Sprint without affecting the Sprint Goal.” – Scrum Guide

Since they are working in the complex domain, things may turn out to be different then the team expected. If they commit to delivering everything in the Sprint Backlog, how would they negotiate changes to it in order to adapt? Flexibility is necessary in order to adapt. If the Sprint Goal was every Product Backlog Item in the Sprint Backlog, why would you need it? The Sprint Goal should be written to provide focus on the most important items in the Sprint. This allows for the ability to negotiate changes to the Sprint Backlog, take on unknown work that emerges during the Sprint and provides the flexibility to adapt.

Increment Commitment – The Definition of Done

“The Definition of Done is a formal description of the state of the Increment when it meets the quality measures required for the product.” – Scrum Guide

Scrum states that quality does not decrease during the Sprint and if a Sprint Backlog Item does not meet the Definition of Done, it cannot be part of the increment. Scrum's commitment is to quality, not scope.

Scrum is Founded on Empiricism

Scrum is founded on Empiricism which tells us we should base our decisions on what is known, not the unknown complexity ahead of us. By committing to scope (the Sprint Backlog), there is an implied expectation that we know everything about the complex product we are developing and therefore, everything will be delivered “on time” at the end of the Sprint. Software resides in the complex domain. We’ll never know everything up front. We have to figure it out as we go.

Scrum Relies on Inspection and Adaption

Scrum relies on Inspection and Adaption to address problems that are uncovered when developing complex products. History tells us that by committing to scope, we lose the ability to adapt and tend to take on tech debt to meet our commitments which leads to a compromise in quality and a less valuable product. 

Scrum is Focused on Quality and Delivering Value, Not Output

Decades of experience and data have shown us that when push came to shove and a Waterfall project pushed to keep its commitments to scope, quality suffered. Why else would you need a “production support team”? Although no one ever really acknowledged what was happening, quality was being sacrificed so scope could be guaranteed. Scrum makes it clear that quality will not be sacrificed. Instead, if necessary, Scrum sacrifices scope in order to guarantee quality.

Velocity and Relative Sizing

You will not find the word estimate in the Scrum Guide. The Scrum Guide only references sizing one time when describing Product Backlog refinement. Although Scrum hints at it through it's word choice, Scrum does not prescribe relative sizing, but Agile practices encourage relative sizing to estimate size of Product Backlog Items. Since the complex domain is unpredictable, it doesn’t make sense to try to analyze a complex problem to create an “exact” or absolute estimate.

The Scrum Guide does not use the word velocity either. Velocity has become a common way to understand the average amount of work turned into a done, usable increment during a Sprint.

If you are using velocity, which is an average, and relative sizing for estimating, which has no direct relationship to time, and you’re working in the Complex domain where you know new work will likely emerge, how can you commit to delivering everything in the Sprint Backlog with any degree of confidence?

Closing

The Scrum Guide is written with intent. Some things aren’t mentioned, while in the content it does describe, the words are carefully chosen with purpose. Scrum is a framework and the wording in the Scrum Guide allows for flexibility, adaption, negotiation and the ability to manage new work that emerges in the development of complex products. It provides a foundation for agility.

If you are expecting teams to commit to delivering everything in the Sprint Backlog, you are more focused on output instead of value and quality. Forcing the team to commit will not magically make work in the complex domain easier. By forecasting, the team can focus on quality, value, adapting, continuous improvement and satisfying the customer by delivering a quality increment at the end of every Sprint. As the team matures, they will become more predictable at forecasting the work they can get done in a Sprint.