Software planning: Why your developers are terrible at estimation

Asking for software development estimates isn’t unreasonable - it’s just impossible.

If you’re a software developer you know what it’s like being asked to guess how long it will take to build something you’ve never built before and only have a vague notion of. You know how difficult realistic estimates are but the manager, boss, or project lead who’s insisting on hard deadlines just won’t relent. They don’t understand - they’re demanding to know when you’ll be done. Bear with them. They aren’t being malicious - they genuinely don’t understand what software estimation is like and what they’re asking you to do: namely, divine the future.

If you’re not a developer but manage or hire them, keep reading for a unique perspective on why software estimates are so hard.

Software developers solve problems for money. You hope the problems they solve will reduce your costs and increase your revenue. It’s not unreasonable, then, to ask “How long will this problem take to solve?” and “How much will it cost?”. Unfortunately, these perfectly reasonable questions are virtually impossible to answer due to many unknowns. When asked, your devs will squirm and ask you questions you don’t have the answer to. Eventually, if you keep pressing them for a deadline and they’re smart, they’ll give you one that’s heavily padded (double it and add 30% is a common rule). In your position, this can be really frustrating. After all, all you’re asking is for them to build some software and that’s what they do for a living - why are they so bad at estimating how long it will take? Allow me to show you.

I want you to imagine that you work full time as a developer - as a creator of stuff and a solver of problems. Let’s make it interesting and instead of talking about software let’s imagine your boss wants you to design and build a steam spider. A steam spider is a real, physical, steam powered robot. It’s like a locomotive but with legs like a spider. The photo of the one above came from the awesome site http://www.steamspider.com where such things are actually built. I’ve used this example with many audiences and it helps get into the mindset of what it’s like to be a software developer.

The steam spider project manager has no idea what the budget for something like this should be so she comes to you, the resident technical expert, for guidance. She thinks you must be knowledgeable about building steam spiders because, after all, you’re the technical expert - the expectation is that you know about everything that sounds vaguely technical and hard to do. “How long will it take to build a steam spider”?, she asks. “I have no idea!”, you might say, “I’ve never built a steam spider!”.

“Well that’s a dumb example.”, you say. “I’m not a programmer and even if I was I sure wouldn’t know anything about building steam engines”. Okay - let’s imagine further that you’re a pro in mechanics and fabrication. You’re great at welding and you’re so good at machining parts you’re taking side jobs on the weekends to fill orders. You’ve even done some steam engine construction and have most of the skills needed for a job like this.

Do those skills help? To build it - sure they help! But unfortunately you’d still be guessing at the estimates for making a steam spider. Just because you have skills in all the basic steps of building something complicated, doesn’t mean it’s possible to give an accurate estimate of how long both the design and implementation will take. There are a ton of unknowns to contend with. How much energy would be needed to make the spider run? How big should the steam tank be? Will it need to be welded together or will fasteners do? What sort of metal should it be made of? How exactly will the legs work? Will we need to build a computer model of it for testing? These questions can only be answered through research and experimentation.

We haven’t even addressed missing requirements from the client: How big do they want the steam spider? How long will it need to run? How much should it weigh? Does it need to carry stuff? Will it need to turn or can it just travel in a straight line? What safety features need to be built in?

“Those are just details”, you’re told. “We must tell the client how long this will take. I need an estimate by the end of the day.” From prior experience you know these “estimates” immediately become “deadlines”. What would you do? You could try to explain that there are so many unknowns a realistic estimate is impossible, but you’ve tried that in the past and nobody listens. If you have to give an estimate and set a deadline for yourself how much time would you allow? Is two weeks enough? What about a month? What about three months or six? If you thought you could do it in two months would you allow for some additional time for all the unknowns? How stressed would you be in coming up with estimates? How stressed would you be after committing to a deadline with so many unknowns? How stressed would you be when you said “three months” but were told “No, the client needs it in six weeks - get it done by then.”

Building software is exactly the same, except I would wager that a modest sized computer program is orders of magnitude more complex than the steam spider we’re discussing. Your developers know how to build websites, databases, business logic, reports, and so on. However, each time you ask them to build something they’re doing it for the first time. It’s like building a steam spider - if you’ve never designed and built exactly that machine before you’ll just be guessing at how long it’ll take. Developers never get asked to write the exact same program twice - that’s what copying and pasting is for.

Deadlines aren’t going away, but appreciate what they cost.

I’m not going on a crusade against deadlines here. Deadlines are real and are usually externally imposed. The client really does need to know when they can use the new system. Marketing needs to know when they can put details of what you’re building on the website. Training needs to know when to create curriculum for the new features. Deadlines are a reality and they aren’t inherently negative. In fact, there’s strong evidence that deadlines help us achieve better performance.

Better estimation was the focus of software development methodology up until the 2000’s. My goal in writing this is to help you understand the cost associated with asking “When will you be done?” from your development team. You should expect that estimates you get will be padded due to the complexity and unknowns of what you’re asking for. The quality of your product will plummet as the deadline approaches and the team races to build features as quickly as possible instead of testing as they know they should. Your team will probably feel some level of guilt for this as they accumulate technical debt and will be less engaged for it.

If the lurking unknowns your team discovers slow down the work, they’ll be stressed out, unhappy, and more likely to heavily pad the next estimate you ask them for. This is vital to understand: most teams are punished if they’re late either by having to work weekends or evenings or being shamed by not meeting their deadline. However, these same teams aren’t rewarded if they deliver early. So, minimizing their risk by heavily padding estimates is exactly what you should expect them to do. It’s what you’re unintentionally incentivizing them to do by making it in their best interests.

So why is estimating software so hard?

Fine, so estimating software is very difficult, but why? Here are some reasons:

• Software is bewilderingly complex and this complexity is always underestimated. Most modern software development methodologies and technologies focus specifically on complexity management and attempting to make things more simple at every turn. Despite this, the complexity of even modestly sized programs means requirements are always incomplete and this makes creating realistic estimates very difficult.
   
• In discussions around project estimation, the Dunning-Kruger effect is frequently a factor. Most non-technical stakeholders genuinely don’t know what is involved in creating new software. Because they have no frame of reference they tend to assume it’s easy. This is why I like using the steam spider example - it helps to have something obviously complex, but not so abstract we don’t have a frame of reference.
   
• Software development is not manufacturing. Each project is unique and falls into the realm of R&D much more than it does operations. Building software products is like designing vehicles - one project might be a car, the next a boat, another a helicopter. It’s not like Ford assembling a sedan. Each project is a new invention.
   
• Power disparity usually exists between the stakeholders paying for the software and the developers who are being tasked to build it. Due to this disparity in power, developers can be pressured into over promising and under estimating. If left unchecked, this will result in terrible software quality, burned out developers, a missed deadline, and perhaps a failed project.
   
• Another factor is the ego involvement developers may have with their work. I’m careful to explain to every developer I work with that “You are not your code.” Most software developers genuinely want to help the enterprise and they know delivering quickly will make everyone happy. They get an ego boost from “saving the day” or being “rockstars”. This leads to underestimation and usually results in bad quality and a burned out development team.  

Is agile the solution?

The 2000’s saw a refreshing change in software estimation: the industry as a whole began to throw in the towel and give up on software estimation altogether. Finally, after decades of excuses and failed requirements processes, what everyone knew to be true was accepted: realistic software estimation is impossible. Agile development methodologies like scrum were widely embraced and the result was better software, faster, and with more engaged clients and development teams. 

There’s no such thing as a panacea, though. We’re still struggling to square agile development methodologies with the need for detailed long term planning and deadlines. Many companies have made the move to “go agile” only to fail because they didn’t change the way commitments and deadlines were thought about at a corporate level. This isn’t just a software problem - it’s a culture problem. A great video to watch on this subject is Atlassian's “Stop going agile”.

What to do differently

The next time you ask for time estimates, remember the steam spider. Appreciate the complexity that lies within all software projects and empathize with your development team. They can no more tell you exactly when they’ll be done and what problems they will encounter than you could have said how long it would take to build a steam powered arachnid. 

Be mindful of the power disparity that may be in play between you and your development team. Pressuring them to finish a project early may compromise the long term success of both the product and the company. Accept that while you may not have a clue how to build what’s required, if the experts you’re relying on say it’s very hard it probably is. Don’t let them over commit - defend their free time and be supportive of a healthy work life balance. If you let them burn themselves out, your best developers will disengage and may just go elsewhere to work.

Consider breaking projects into small phases and only seeking estimates on the first phase. If this works for you, consider learning more about agile development and take an honest assessment as to whether or not your company and clients could support such a strategy.