99 Bugs In The Code

A week or so ago I saw a tee-shirt with the following slogan:

99 little bugs in the code
99 little bugs in the code
Take one down, run it again
117 little bugs in the code.

This is amusing because it speaks to an experience that every developer has encountered; you make a change and suddenly stuff that had previously been working is now broken. It is for that reason that just testing a change is not sufficient, you need to test all of the existing functionality too. The problem is that this means performing a the same tests over and over again. This is both dull and time-consuming, so developers will often skimp on these regression tests.

If only software engineers had access to some type of device that is good at running the same tasks repeatedly. There is a little sarcasm in the preceding statement, but many engineers do not write tests for the computer to run. Having automated testing of code allows an engineer to quickly verify after each change that no bugs have been introduced and that existing functionality works as it did before. Failing to have these automated tests in place is all but guaranteeing that in the near future a change will introduce a new bug that is not detected until it is too late and the defect is in production. The rest of this article discusses how to ensure that as developers write new code they also write code to test it.

Ensure that writing and running the tests is as straight-forward as possible. There are frameworks available for major languages that aid in this process. Reach agreement with the engineering team on which one will work best for you. If possible ensure that the tests can be run individually from within the IDE, this enables an engineer to quickly and easily run tests for just the class or method that they have changed. For instance, I am creating an application in Ruby on Rails (RoR). RoR comes with a testing framework built in, called Minitest. Rails will create a test class for each class added to the code. From within Atom, (My code editor of choice) a simple key press will run the current test class, or run the last set of tests again. The output from the tests appears in the editor, so I do not need to switch programs at all whilst making changes and verifying those changes.

Have a requirement that tests are run at some stage before code is pushed into the source control master. Depending on your workflow that might mean running tests before each commit to source control, or only when merging to master. Source control software has hooks that enable this policy to be enforced and the commit or merge blocked if there are failing tests. Where you place this requirement depends on how long your complete set of tests take to run. If they execute fairly quickly, having the full set run before every commit is practical. If they take a long time to run, you may find them slowing down commits too much.

You also need to measure how well these tests are testing your code. There are tools that will measure the coverage of your test suite; ie how many of the lines of code are being run by your tests. Returning to my current example, there is a plugin for Minitest that reports the percentage of code covered in the most recent run whenever the suite of tests is run. The plugin also generates a report that breaks the coverage down to class by class and then allows you to look at the lines of code that were not executed. This enables you to right additional tests. Note that 100% code coverage is not the same as testing that your code is bug-free. For example look at the following code:

def serve_drink(age)
  return true if age > 21
  return false
end

You can obtain 100% coverage of this code with the following couple of tests

test 'Ok to serve drink to adult' do
  assert serve_drink(42)
end

test 'not ok to serve underage drinkers' do
  assert_not serve_drink(18)
end

However, the tests have not found the bug in the code, which is that the code will return false if asked if it is ok to serve a drink to someone who is 21. Unit tests are also not going to catch that the code as written is limited to use in the US where the legal age is 21. Unit tests do not ensure you have bug free code, but they do ensure that code is still working as well as it did before the change.

Code reviews should also review the test written, this helps ensure that the tests have been written and can also catch errors in the tests themselves. In the above example adding a third test for the boundary condition (age equal to 21) would catch the bug.

When a bug is found in QA or production, in addition to fixing the bug the developer should write new unit tests to catch that bug should it reappear in future. Having a bug in production once is a misfortune, having the same bug appear a second time looks like carelessness.

In summary; make writing tests part of the development process, make it as simple as possible to write and run them, measure the coverage, and keep adding new tests as new defects surface.