#NoEstimates Does Not Mean “No Estimates”

Have you ever given an estimate that turned out to be way off? No need to answer that question — the answer will be “yes.” More likely than not, you work on teams that are regularly asked for estimates, regularly invest effort estimating, and regularly get it wrong. This inevitably leads to a retrospective action along the lines of “we must get better at estimating!” Unfortunately, I don’t see many teams get better at estimating, but I do see this retrospection action coming back iteration after iteration, project after project. What can we do about it?

Before we try to fix something that seems broken, it is usually worth taking the time to ask ourselves “what are we trying to achieve anyway?” If we ask this question about the role of estimates in software development, we discover that it depends upon who is asking the question.

• The customer uses the estimate as input to their planning process. They want to know how much the project is going to cost (in time and/or money), so that they can compare it to the likely returns before making a commitment. They may use the estimates to help trim the project scope to fit within a fixed budget. These activities are “forecasts.”

• The management team compares the team’s actual performance against the estimate throughout the life of the project to check that everything is going to plan. If things seem to be slipping, then they can take remedial action: changing scope, adding resources, extending timescales. These are “management” activities.

• The development team has very little direct need for estimates. The benefit they get is a secondary one — they need to understand the tasks before they can make an estimate, so initial analysis often happens during estimation. This is an “analysis” activity.

Rather than try to “get better at estimating,” let’s see if there are other ways of satisfying the needs of each of these groups.

Customer (Product Owner)

The customer is trying to get the most return from a fixed investment. There is always more to do than there are resources to do it, so project estimates are used to try and balance the supply versus demand equation. An estimate on its own is not enough — the customer will also want to consider the expected benefit of the project — and it’s at this point that the insidious nature of estimates shows itself clearly.

An estimate is just a guess. If made scientifically, we would hope that 50 percent of estimates were over-estimates, while 50 percent were under-estimates. What we find is that almost all estimates are under-estimates, due to the generally optimistic nature of technical organizations combined with external pressure to keep them small. The forecasted project benefits are similarly optimistic, but are generally not subjected to the same level of scrutiny post-delivery.

These two tendencies combine to make portfolio decisions very risky. Instead of investing more scarce resources in generating better estimates, it’s time to consider other approaches. Smaller, safe-to-fail prototypes is one possibility that has been successful for many organizations. These have been described extensively in the Impact Mapping, Beyond Budgeting, and Lean Startup books, as well as elsewhere. Another useful technique that is successful in some contexts is Cost Of Delay, which applies a financial model to prioritization decisions.

Management (Program Office, Project Manager, Scrum Master)

The management team needs to ensure that the development team delivers the expected value to the customer. Part of this is the traditional job of making sure that the right resources are available. There’s also the critical need to identify and manage the risks that are inherent in any software project, which is where “old school” management techniques can sometimes rise to the surface. See Waltzing With Bears.

The Scrum methodology uses story point estimates and velocity to track delivery. When the actual burndown deviates from the estimate, I’ve heard managers ask for teams to work longer hours, deliver more story points, or weaken the definition of “Done.” All of these will lead to a poorer quality delivery, that’s more expensive in the end — an outcome which is rarely justified by the impact of shipping an extra story earlier. Similar stories come from non-agile development organizations as has been extensively document in The Mythical Man-month and Death March, among others.

Instead of trying to get more out of the development machine when things start to go wrong, it’s much more effective to try and discover the unknowns earlier in the process. Once the risks have been identified, it’s much easier to take actions to mitigate them before they become a problem. An approach that I have found very useful is using Liz Keogh’s lightweight complexity scale. This can help us identify the areas of our product that we know the least about, and which we need to investigate early.

It’s at this point that the textbook approach to prioritization begins to cause problems. If the customer has prioritized some work to be done, but the team knows there is significant risk elsewhere, who gets to decide which gets done first? This is when a good manager can facilitate collaborative conversations that lead to the best outcome for everyone. It’s in these conversations that the team will discover the minimal, coherent slices through the product, allowing early delivery of functionality, and early feedback.

Estimates can lead management to focus on the numbers, applying deterministic techniques that optimize for efficiency and utilization, rather than identifying and exploring the team’s areas of ignorance.

Development

The development team needs to deliver value to the customer, but crucially, they need to know what the customer considers valuable. In all too many organizations, there is a belief that a well-written requirement or user story is all that the development team needs. It’s good if everyone understands that estimates based on analyzing these requirements are unreliable, but it should be understood that the analysis achieved in this will also be shallow. Without active collaboration between customer and development many risks, ambiguities, and inconsistencies will lurk until late in the development process.

Various methodologies have been developed to help guide and record the process of “deliberate discovery.” Particularly worthy of note are the lightweight User Story Mapping and Behaviour Driven Development. Neither of these approaches prevents the creation of estimates, but they don’t require them either. Indeed they are orthogonal to estimation and their conflation within many agile methods has been called “agile’s trojan horse.”

#NoEstimates

The #NoEstimates hashtag first saw light of day on Twitter a few years ago. It was championed initially by Woody Zuill, and then by Vasco Duarte, Neil Killick, and others. You can read the posts and tweets, but to get the most fun out of the whole proceedings you should read the comments, which mostly go something like: “How could we develop software without estimates ?!?!?”

The idea of doing something without making an estimate first is not a new one. In the domain of civil engineering, recall the pyramids or any of the great European cathedrals. In software development, there are 2 precursors to the current #NoEstimates discussion that are instructive to consider, “naked” planning and probabilistic planning.

Probabilistic planning is a formalization of a common-sense method — if painting the fence took you 3 hours last time, it will probably take you about 3 hours next time. In software development we’re painting a different fence each time, so we mine historical data about how long we took to do similar pieces of work to forecast how long the current piece of work may take. Clearly, data quality will be crucial, as well as the ability to accurately classify and compare different work items.

By combining this historical analysis with some elements from queuing theory, we can cheaply and reliably forecast schedules and manage risks. David Anderson claims that “a probabilistic forecast made in this way is likely to be significantly more accurate — more similar to the actual outcome — and is also significantly faster and cheaper to create than a traditional deterministic plan that requests an estimate for each work item within the project scope.” See Forecasting Projects in Kanban.

This approach may work well if suitable historical data and the statistical skills to analyze it are available, but it is often beyond the capability of smaller, younger organizations. It was such a situation that gave rise to “naked” planning, which came about when Arlo Belshee and his team applied ideas from “Lean” to their implementation of Scrum. See Naked Planning Overview.

They specifically addressed the waste that they observed when estimation used valuable resources, but did not deliver any visible value. The process that they developed replaced Scrum’s prioritized backlog of sized stories with a fixed queue of 7 minimal marketable features (MMF). The MMFs are similar to User Stories, but no estimation has been done on them. The product owner chooses the content and order of the queue and can change this at will.

To give the management team some insight as to what will get delivered, each MMF is dated when it enters the queue and again when it is delivered. Over time they calculate an average time to deliver an MMF and call this the “Disneyland wait time” as in “Your expected wait time today from adding an MMF to the queue is approximately x days.”

Arlo makes an interesting observation that, not only does this save on estimation, but it also delivers better outcomes, because “in software, you get better results by prioritizing based solely on value than you get by including cost. This is because software is a highly-leveraged domain where cost is distributed linearly and value is distributed exponentially.” See Naked Planning Explained — Kanban in the Small.

Conclusion

All estimates are guesses. The sort of estimate most software development teams make today take considerable effort to produce and yet the main benefit is actually derived not from the estimate itself, but from the act of analyzing the requirements to produce the estimate. Other needs that are traditionally satisfied by estimates can be addressed using other approaches that are less expensive and more clearly express the project’s risks.

The “#NoEstimates movement” has many proponents, but at the heart of it isn’t a petty rebellion against estimation, but a desire to provide better value than estimates currently do. If the estimates your team produces work for your organization, then all is well. If, however, estimates seem to suck up much needed resources without helping your organization succeed, then look for more effective ways to provide the information that is needed.

When you repeatedly find yourself trying to “get better at estimating” you may be in a “cycle of continuous no-improvement.” Take a step back and ask yourself what you’re actually trying to achieve, and look for a better way of achieving it.

--------------------------------

Seb Rose is the lead author of the new Behaviour Driven Development (BDD) book from the Pragmatic Programmers: "The Cucumber for Java Book".

He blogs at Claysnow and tweets as @sebrose