Limitations of Code Coverage

Previously, I modified the Stack Project to correct errors identified by Checkstyle, PMD, FindBugs and JUnit. Then, I added some unit tests to achieve 100% code coverage as reported by Emma. This time, I was challenged to introduce an error to the program while maintaining complete code coverage.

Method 1

My programming partner for this week, Ronn Reeves, came up with the idea of adding a Stack.size method that always returns one as the size of the stack. We would achieve total code coverage by adding a test case that would test the Stack.size method on a stack containing exactly one object. To demonstrate that there is an error in the program, we would test the Stack.size method on a stack containing any number of objects besides one. This is a perfectly good example of a system that has 100% code coverage, but still has a blatant error. However, I wanted to try a subtler approach.

Method 2

Following a hint given in class by Professor Johnson, I modified the program so that the Stack.push method works only as long as the current size of the stack is less than some arbitrary maximum--in this case, five. Any further attempts to push objects onto the stack are ignored without any warnings. My existing tests still achieve complete code coverage because none of them push more than five objects onto a stack. In order to expose the error, I created a new test that attempts to push six objects onto a stack, and then pops one object off. Since the sixth object is never actually pushed onto the stack, the fifth object gets popped off instead; since the fifth object is not the same as the sixth object, the test reports an error.

My Stack Project with 100% coverage and an introduced bug is available here

Conclusions

Last time, I concluded that 100% code coverage is not necessarily a worthwhile goal. This time, I can see that 100% code coverage could lead to a false sense that one's code is well-tested. As this exercise clearly demonstrates, however, code coverage is not a measure of code correctness. It was very easy coming up with code that achieved 100% code coverage and yet still contained significant bugs. If high code coverage is not an indicator of well-tested code even in such a simple system as the Stack Project, then that goes doubly so for a more complex system where the bugs are potentially more subtle.

Code coverage tools cannot help you write better tests for your system, and they cannot identify errors (or errors of omission) in your code. They can only point out areas of code that are untested or under-tested. Nevertheless, they remain useful testing tools because they highlight areas of code where bugs may be hiding.