Code coverage is a tool, not a metric of quality

My goal with this simple example is to demonstrate that using code coverage as a metric can lead to the false scene of quality.

I have been working over the last few months to increase the quality of unit tests in a project I am working on. The developer that originally wrote a good deal of code did not provide unit tests and we want to make sure we are not introducing bugs as we change things. For me a unit test is a contract and proof that you are doing what you think you are. When I write a unit test it tells others: “This is my expected behavior. If for some reason this test fails, it is because the code base has changed or was written in such a way that will break behavior I expect elsewhere or within the unit.” In various projects I have worked with each team puts a different emphasis on the importance of unit testing or testing in general. Some teams/projects, when against a hard deadline, will cut back on the amount of testing they do. Others will add qualifiers to their DOD, definition of done, such as: "We will not allow you to check code into the repository unless you have 80% code coverage". I feel that the expectation of code coverage causes developers to miss the goal. Discover bugs sooner and protect against harmful changes.

Code coverage is a good tool to help you understand what you are covering in your tests but should not be the end all be all of quality. The following examples will show what bugs might be missed in a 100% covered situation. This Example is based on some code I found not long ago.

As you can see I have hit every decision point. This class has 100% coverage. Now imagine there are 20-30+ additional enums and you can't see in one shot the typo in EXAMPLE_TWELVE or that I added an EXAMPLE_1 that has the same value as EXAMPLE_ONE.

Lets take a look at the Unit tests that generated the above coverage.

As you can see we really only tested around "EXAMPLE_ONE". This test tells us nothing about the other enums. Next if you look at getEnumByValue(long value) you will notice that it takes a "long" but value is only an "int". While its true that a unit test is not likely to find the "long" issue it just goes to show you that 100% coverage does not mean free of bugs.

In the following example I am going to use a JUnit Runner called Spockito to help write unit tests. Spockito allow you to reuse code and put your common values in a table. I feel that this makes it easier to add cases later and easily read what is going on.

Looking at the above tests we found 3 failing points not discovered by the previous test. Again we have 100% code coverage but now we have some value in our unit tests. First it exposed the typo in EXAMPLE_TWELVE, it reports "expected:<EXAMPLE_TWELVE> but was:<null>". Next we see that there is an issue with EXAMPLE_1, it reports "expected:<EXAMPLE_1> but was:<EXAMPLE_ONE>". In the code this example was derived from two fields could have the same value but the way it was implemented you could never get the second value back. Lastly again back to Example_TWELVE we see: "expected:<12> but was:<1>".

In summary I would like to say that code coverage is an amazing tools. I use it whenever I am writing unit tests to help me see blocks of code that I might be missing entirely but... It is important to realize that just because you covered a block of code your test may not add value.