Measure Time to Detect Defects

When I was young I read a study by TRW done in the ‘60s that measured the cost of fixing a defect that was detected at different points in the development cycle. If the developer who wrote the defect found it immediately after writing it then we can assign one unit of effort for resolving that defect. If that’s true then it takes something like seven units of effort to resolve the defect during the testing phase and 15 units of effort to resolve the defect just before release and a whopping 67 units of effort to resolve the defect if it makes it into the hands of the customer. In other words, the cost of fixing defects grows exponentially with the amount of time it took from when the defect was created to when the defect was resolved.

It’s always cheaper to fix defects sooner. This is because the longer we wait from when a defect was created the less we are familiar with it. The vast majority of time and effort in debugging isn’t involved in fixing the defect. Some defects do require a significant amount of reengineering but most defects are small problems that can be quickly and easily fixed. What can’t be done quickly and easily often is finding of the defect in the first place. Much of the time, finding a defect is where we spend most of our effort and once we locate the defect it’s usually trivial to fix.

My friend Llewelyn Falco likes to put it this way, being told that there’s a defect in a program is like being told that there is a misspelled word in the dictionary. Fixing a misspelled word is easy but finding it in the dictionary can be tedious and time-consuming. This is what debugging is often like so having ways to help us find defects and make defects more findable can be a great asset to a program.

It is almost universally true that the sooner we can find defects the more straightforward and cost-effective it is to resolve them. Because of this I pay a lot of attention to finding defects as early as possible. I find that having a good suite of unit tests can help immensely with this and this is one of the reasons that I’m a big advocate of doing test-first development, because it gives us a set of regression tests that we can use to find defects and validate our software is working as we expected.

Of course, regardless of how quickly you find defects, it’s even cheaper if you don’t create them in the first place and I find that doing test-first development helps me eliminate a huge range of defects that might show up in my code otherwise. These range from fat-finger typing to conceptual and logical mistakes. 

My unit tests are my sanity check and I find that I can build far more stable and dependable code with far less effort using test-driven development. Like any discipline, doing TDD requires skills and it’s very possible to do it incorrectly. Doing TDD incorrectly has very little value, just like doing any activity incorrectly has very little value. But when we do test-first development well, by building good, unique, implementation-independent tests that support refactoring and validate that our features are working as we expect them to, then this gives us a tremendous amount of confidence in our code and allows us to work fast.

Note: This blog post is based on a section in my book, Beyond Legacy Code: Nine Practices to Extend the Life (and Value) of Your Software called Seven Strategies for Measuring Software Development.