Tracking Project Metrics

How do you track the health of your software projects? Ideally you could come up with few, easy to collect metrics and have your Continuous Integration system generate the information and maybe even graph it over time. What we track is going to be based on a set of beliefs and assumptions, so I think we should make that clear.

My Software Metrics Beliefs and Assumptions

• The simplest code that solves the problem is the best. Simple does not mean rote or repetitive. It means well designed, well abstracted, well factored.
• Unit Testing improves the quality of code.
• Overly complex code, code that is not well factored, "big" code is hard to unit test.
• The metrics need to be easy to interpret and easy to gather or I won't do it.

Based on those beliefs and assumptions we have defined the kinds of things we care about. We want simple, small classes and methods. We want classes that fit the Single Responsibility Principle. We want unit test coverage. And we want to know when we deviate from those things.

Inevitably this won't tell you the whole picture of a project. Some deviation is inevitable as well (we're not perfect). But this is giving us a picture into a project that would let us look at "hot spots" and determine if they are things we want to deal with. It will never tell you if the system does what a user really wants. It will never fully tell you if a project will be successful.

The Metrics I Came Up With

• Unit Test Coverage - How many code paths are being exercised in our tests.
• Cyclomatic Complexity - Number of methods over 10, 20, 40
• Lines of Code - General size information
• Methods over a certain size - Number of methods over 15, 30, 45 lines
• Classes over a certain size - Number of classes over 150, 300, 600 lines
• Afferent/Efferent Coupling - Dead code and code that does too much

Tools

FxCop

For Example to calculate Cyclomatic Complexity you can implement a class that extends BaseIntrospectionRule and overrides the VistBranch and VisitSwitchInstruction.

public override void VisitBranch(Branch branch)
{
    if (branch.Condition != null)
    {
         Level ++;
     }
     base.VisitBranch(branch);
 }</p>
<p>public override void VisitSwitchInstruction(SwitchInstruction switchInstruction)
{
    Level += switchInstruction.Targets.Count;
    base.VisitSwitchInstruction(switchInstruction);
}

Coverage

NCover is a proprietary coverage tool. High on wow factor, but expensive. NCover was once free and open source and gained a good following, but they switched and closed it and made the decision to charge $150 - $300 a license for it depending on the version.

(I lament the state of the .NET ecosystem with regard to tooling. Either an OK open source version or an expensive commercial version is not a real choice. There are so many good options in the Unit Testing space, but not in the coverage space.)

NDepend Mini-Review

One of the tools that I've been looking at for .NET projects is NDepend. It seems to cover all of the cases that I mentioned above except for the Code Coverage case (although it integrates with NCover, but I haven't looked at that). It has a really cool query languages, that looks a lot like SQL, that lets you customize any of the existing metrics that it tracks or write your own (it comes with so many that it seems like I just customize). It comes with so many metrics that in practice it can be seem quite overwhelming. I think the right thing to do for most projects is to pick the handful that you care about and limit it to that.

NDepend comes with NAnt and MSBuild tasks that will let you integrate it into your build automation. It also comes with an XSL stylesheet to integrate the NDepend output into CruiseControl.NET for reporting purposes.

Some things you might run into:

• C# support is the primary language (e.g. No support for code level Cyclomatic Complexity for non-C# language, but still supports IL level CC). Only a problem if you are forced to do VB.NET of course, or use another CLR language.
• A single license is $400 and 20 licenses or more are over $200 each. So price could be a concern to many people. I'm thinking it might make the most sense to run this on a CI server, so in that case you would probably only need a small number of licenses.
• It integrates with code coverage tools, but currently only NCover. See the previous comments about the cost of NCover combined with the cost of NDepend if price is an issue.

What Else?