Dealing with Error and Bias in Estimating

In a previous article (here) I made the case that in Geographically Dispersed Teams, even those following an Agile model of working, it will be increasingly necessary for a formalised Team Schedule to be used to coordinate work, and that team members would need to be skilled in Scheduling and Estimating. This article addresses the latter. 

Scheduling is the process of sequencing tasks and assigning time and resource to them. A schedule provides performs three functions; to facilitate the coordination of the application of people and resource, to provide a baseline to monitor performance and respond to change, and to support governance by allowing stakeholders to monitor the expenditure and delivery. Whatever scheduling method is adopted, effective estimation of the time and resource requirements for each task are critical. If the estimates used are “poor”, that is they deviate dramatically from actual values, it becomes impossible to determine if the variation from the schedule is caused by poor performance or poor planning; this means that the coordination, change-control and governance functions all become difficult.

There are two basic reasons that Estimates are poor: “Error” and “Bias”. “Error” covers the effects generated by policy and procedure; “Bias” covers the psychological effects that can drive the Estimating Process astray. 

Error in Estimating

There are several ways that the estimating policies applied, and the procedures adopted, can cause individuals and teams to generate poor estimates. One is the estimators’ failure to understand and deal effectively with insufficient or inaccurate information about the programme of work to be followed and/or the capabilities of those performing the work. This is likely to be extremely high at the start of a programme but will hopefully improve as the programme progresses; this effect is called the “Cone of Uncertainty”. The source of estimating error here is not the uncertainty, this is inevitable, it is the failure to explicitly manage that fact that any values generated will be subject to this inevitable variation and so that all should be expressed as a range reflecting the position of the programme in the Cone.

This is linked to the notion of “Unwarranted Precision”; there is a general tendency in estimation to treat “Precision” and “Accuracy” as synonyms when it comes to the value generated: they are in fact antonyms, opposites. The more precise (exact) an estimated value, the less likely it is to be accurate (correct), particularly in the early stages of the cone. Range calibration training is usually required to teach estimators how to place correct ranges on their estimates; wide enough to be accurate, precise enough to be useful.

Another source of Error is a misunderstanding by estimators of the effect of the process they use to generate the values. An estimate is necessary when measuring the actual value is unwarranted or impossible; with planning estimates it is the latter because the actual values are in the future! The two most common approaches to generating estimated values are also the least inaccurate; “Comparison” and “Judgement”. Comparison relies on using the actual values from an equivalent task in the past, but this relies on extensive and accurately recorded historical data to be useful. Judgement by experts is the least accurate and most prone to Bias effects described below. Even though the measurement of the actual value is impossible, “Measurement” is still the most accurate approach to defining estimates. The key is to determine the appropriate proxy measure, for example a decorator measuring the area of the floor of a room to estimate the amount of time and paint that will be needed to decorate it. This proxy measure is called a “parameter” and the technique is called “Parametric Estimating”. Defining the appropriate parameters and measuring them accurately is an area where most teams need guidance and development.

A third source of Error is generated by how estimates are used; are the values generated then used as targets to assess performance? When this happens, the estimating process becomes subject to “Campbell’s Law”, “the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor”. Using estimates as the basis of targets will lead to a distortion of the estimating process, leading to poorer estimates and, in turn, failure in the planning and control process. It is important to develop a team culture that values the accuracy of estimates over the fear of assessment.

Bias in Estimating

As well as explicit procedural problems in generating poor estimates, there are also implicit psychological ones; there is considerable evidence that estimators are subject to an effect called “The Planning Fallacy” which is a manifestation of a human behaviour pattern called “Optimism Bias”. Optimism Bias is a systematic pattern of deviation from rationality in judgement that causes someone to believe that they themselves are less likely to experience a negative event; it is captured nicely in Tom Demarco’s humorous quote, “An estimate is the most optimistic prediction that has a non-zero probability of coming true”. There are a number of psychological processes that contribute to the bias; “Valence Effect” where people tend to overstate the likelihood of attractive, positive outcomes and understate the likelihood of unattractive, negative ones; “Interpersonal Distance” where the greater the perception of the distance between the impact of a negative outcome and its personal effect, the easier it is to dismiss; “Representative Heuristic” where people place too much emphasis on single, big, memorable but unlikely failures and insufficient emphasis on the numerous, small but likely ones; and “Singular Target Focus” where people treat themselves as unique individuals but others as generic categories judging their own performance as better than the average.

“Hofstadter’s Law” succinctly captures the Planning Fallacy; “A task always takes longer than you expect, even when you take into account Hofstadter’s Law”. The fallacy is defined as “the tendency to hold a confident belief that one's own project will proceed as planned, even while knowing that the vast majority of similar projects have run late”. One solution is to train estimators to use “Outside” thinking rather than “Inside” thinking. An “Inside” perspective places emphasis on understanding the task as a unique case and on predicting how it will be performed, constructing a narrative about how the task should be completed, focusing on the intended sequence of events rather than the likely sequence; historical failure is interpreted as the result of unprecedented and unpredictable external forces that are unlikely to recur and so can be discounted. An “Outside” perspective places emphasis on considering the task a one of a class of similar tasks and on measuring how others performed them; list all the things that could go wrong with a task (big and small) and how often they have occurred in the past; this reduces the ease of dismissing the epic fail as unlikely and hence success as more likely.

No one should estimate a task alone, group discussions reduce the effect of individual psychology and experience; techniques such as ‘Delphi’ and ‘Planning Poker’ can facilitate more efficient and effective group estimating. The majority of drivers rate themselves in the top 30% of skilled drivers; they rate other drivers as much less skilled and agree that others would also rate them lower than they rate themselves; estimates should be based on how someone else would perform rather how we feel we would perform ourselves; it is important that individuals do not estimate their own tasks.