Defects: Seeing the Forest for the Trees

Defects.  They go by many names, bugs, issues, undocumented features.  Regardless of what you call them, they’re present in every off-the-shelf (OTS) or custom developed software product on the market today.  As much as we would like to and as hard as we try, it really is impossible to develop defect free computer programs.  Believe me, if it were possible, Microsoft would have figured it out so they could save millions of dollars per year by not having to developing patches & updates.  [Full disclosure – I am a Mac user and we get updates too.]

Where do defects come from:  The simple answer is, our computers are only as good as the instructions we give them.  But  keep in mind that our computers receive their instructions 3rd or 4th hand and not directly from the source.  For example, the requirement provided by the business person that “error text shall display in red” sounds very clear and straight forward.  That requirement is documented by an Analyst who, in turn, works with a Technical Lead who eventually gives that requirement as a larger body of work to a Developer.  While the requirement sounds straight forward and extremely clear, if the business person has a particular shade of red in mind, they may report a defect against the requirement. 

More likely defects are found in the coding logic which is developed to meet the business requirements.  For example, if a business user provides requirements that state, “If the user does not exist in the system, then ask them to create an account”.  Again, this is a very simplistic example, but if a user who did not exist in the system was allowed to access the system, then a defect would be generated against this requirement.

On occasion, when we make enhancements to systems, we inadvertently create a situation where our new code causes an unintended result in some of the existing code.   

Who’s supposed to find the defects:  Let’s go through a very high level and overly simplified testing model which starts with most defects being found by developers during their initial rounds of Unit Testing.  Generally, junior developers will develop solutions that meet the requirement as stated.  More seasoned developers will develop to meet the requirements and will also think through some ‘negative’ scenarios; not only what should happen but what shouldn’t happen.

If you have the development resources, and depending on the project constraints, a developer should submit his solution to a peer for Peer Testing.   In smaller development shops or on smaller projects, this step is often skipped and the solution is passed to the Team [Tech] Lead for testing & code review prior to being formally submitted to the Quality Assurance team for further testing, known generally as Integration Testing, where several functional components are combined into a larger functional solution.

By the time the Quality Assurance (QA) team gets the solution, the majority of logic defects should already have been resolved.  One might think, given the developer’s testing, the QA team could bypass testing the system to ensure the basic requirements were met however that’s their starting point.  After their initial testing of the functional solution, the QA team will generally move on to negative testing, end-to-end system testing, and finally regression testing; all along the way finding and logging defects.

The next set of testers are generally users who are responsible for testing and accepting the system in a process simply known as User Acceptance Testing (UAT).  Many confuse User Acceptance Testing as being the final step in testing before a system is put into production.  In fact, UAT involves users initially testing the smaller functional solutions delivered from the QA team, throughout the project, against real world business scenarios.  Eventually, as more and more of the overall solution is delivered,  UAT testers do test the entire system from end to end, against actual business scenarios.  It is always best to get fully functional sections of your system in front of users as soon as possible to help mitigate a number of issues, one being resource allocation around developing new functionality versus defect resolution.  That said, during UAT users log defects which may be related directly to requirements or may be related to how a requirement was interpreted.       

What happens to the defect once it’s logged:  This is where it often becomes hard to see the forest for the trees.  Remember when I said all systems contain defects?  Defect Resolution is one area where project costs can really get out of control unless the technology and business teams work very closely together to prioritize which defects are resolved.  Throughout all the cycles of testing, defects have been generated and assigned a Priority.  In part, the Priority helps the IT team determine which defects to resolve.  Other attributes of the defect help the IT team determine when a defect should be addressed. 

As a manager, one of the conversations I have with my business partners is that defect resolution is not free.  While we employ a number of methodologies to try and minimize the number of defects the IT team generates as a result of logic errors, misspellings, or misinterpreted requirements, we will inevitably have defects and there is a cost to resolving those defects.

Based on the size of the organization, the Roles performing the next several steps may vary but the process is generally the same.  Once a defect is logged, someone has to review the information within the defect, ensure the test case steps were followed correctly, and make sure the defect is reproducible.  Once the requisite details are satisfied, the defect can be reassigned to the next person in the process.  At some point, the project leadership [PMs, Tech Leads, Analyst and Test Lead] should meet to discuss all open defects and affirm the priority of each.  Ideally this would happen before the defect is assigned back to a developer to be resolved. 

Before the developer begins resolving the defect they should take a few minutes to meet with their Tech Lead or the Analyst to ensure they fully understand the particular defect as well as the expected successful resolution.   Once the developer has resolved the defect, the testing process begins again with unit testing and concludes with UAT.

Even a very simple spelling mistake can create an hour of work when you think about all the steps it takes to review, resolve, and retest the defect.  I have seen projects enter the testing cycle under budget, only to end up over budget as the result of the cost of defect resolution.  With communication, negotiation, and a solid process, you can prevent defect resolution from becoming your Enchanted Forest.