Measuring Software Project Health

There are many indicators of how well a software project is doing. One obvious example would be how many problem issues, such as violations of project standards or deviations from programming best-practices, there are in the code. Another example would be code coverage, which is a measure of how thoroughly the code is being tested by the test suite. A project with high coverage is presumably less likely to have problems. Cyclomatic complexity is a count of the number of linearly independent paths through the source code. The more conditional statements or loops the code has, the more complex it is, the harder it is to test, and the harder it is to understand. Another indicator is coupling, or the extent to which classes in a system are interdependent. A highly coupled project is less flexible because a change to any part of the system could potentially impact many other parts of the system. Code churn is a measure of the amount of change to the system. A project with high churn is in a high state of flux and may be less stable than a project with low churn. There are many other indicators of project health.

None of these indicators by themselves are good measures of the overall well-being of a project. Just because a project has high cyclomatic complexity, for example, does not necessarily mean that the there is a problem; perhaps that is the nature of the problem that the project is trying to solve. However, all of these indicators combined make a pretty good measure of project health. Furthermore, a static view of these indicators is not as informative as observing these trends over time. A project might legitimately have 70% coverage and an average coupling of 12 (i.e., on average, each class depends on 12 other classes). On the other hand, if the project's coverage is declining, or if its coupling is increasing, then this may signify some problems.

Hackystack is an open source framework for collecting, analyzing, visualizing data about software projects. Users configure their software development tools to use "sensors" to send out project data to the Hackystat server, which then presents these data in easy-to-interpret visualizations.

Configuring Our Project for Hackystat

The Hackystat sensor example project's user guide was useful in setting up the environment variables and creating the sensorshell.properties file. I did find it a bit confusing, however, that the Hackystat installation guide did not contain the same information. Configuring Eclipse and our project's Ant targets to send sensor data was pretty easy, especially after seeing how it was done in the stack-johnson project. One caveat was that I had to install the JAXB package, even though I am using Java 6 (on OS X 10.5.5), which should already include its own JAXB 2.0 implementation. I also had to create a UserMap.xml containing my SVN repository and Hackystat account information file in order to get the SVN sensor to send data properly.

Our Project's Health

According to Hackystat, our DueDates project is pretty healthy. The only marginally problematic area is our overall method-level code coverage, which is 64% -- lower than desirable.


A more detailed view of the coverage reveals that this low value is mainly due to an extreme outlier.


Specifically, the DueDates class is untested (!) because we weren't sure how to test it. All the fields and methods in the class are private, except for the main method which is void, so we can't get any data out of the class to test, and it seems "wrong" to change the fields and methods to non-private just for testing purposes. The coverage for the HawaiiStateLibrary class is also somewhat low. We can't test most of the class without logging in, and we didn't want to save login credentials to the repository, and we couldn't pass command line arguments to the test because we are using Hudson.