NCover

From time to time I get questions about why a developer would use code coverage or how best to use it, so I've put together a compilation of my thoughts about code coverage analysis and how it should be done. I'll be writing a series of these articles over time. Let me know if you have ideas for other topics you'd like me to elaborate on.

What is Code Coverage?

Code coverage analysis is a measurement that is taken on running program code reporting how much of that code was executed. Specifically, code coverage usually tells you how many times each line of code was executed. There are various forms of coverage metrics, such as function coverage, sequence point coverage, statement coverage and branch coverage. I'll elaborate on some of these in later posts. For now, NCover supports sequence point (which is basically statement coverage) and branch coverage.

Why do you need Code Coverage?

Code coverage is most valuable as a feedback mechanism for test-driven or agile development methodologies. Both of these methods rely on a developmental feedback loop that promotes the addition of features while maintaining a predictable quality level. Code quality is one of the most important concerns of any software development organization. Rapid development and technical wizardry will do you no good if customers can't rely on the quality of your product.

The way most modern software development teams maintain the quality of their software is by building a suite of unit tests that automate the verification of code correctness. These unit test suites are designed to run through (often thousands) of simulated program functions, verifying that the code is producing the expected output. While creating a number of unit tests is laudable, just creating a suite of tests alone is not enough. After some point, adding more tests does not necessarily mean that the code has better quality. You may just be testing the same code over and over again, while missing an important piece of code entirely. Any bugs in code that is not tested will not be detected by your unit test suite and thus could creep into production.

Coverage analysis closes that feedback loop by reporting on the comprehensiveness of your unit tests. Code coverage reports allow developers to quickly find code that is not executed by the test suite.

When should you use code coverage?

Early and Often...is the short answer. The heart of an iterative development environment is a continuous integration server. Each time a developer checks code into your revision control system (or on a regularly scheduled cycle), the build server should check out the latest code, do a full build of the product, run the test suite and report test success/failure and code coverage metrics. A continuous build process facilitates the early detection of code errors. Immediate reporting of code coverage also makes it clear to everyone on the team if the quality of your tests are increasing or declining.

How do I implement code coverage?

For the .NET framework, of course, I'd recommend NCover as the tool of choice. It's been used all over the world by over 100,000 developers, and has 4 years of maturity behind it. For other development platforms, there are various other code coverage tools available.

NCover performs analysis on your code by instrumenting it behind the scenes. If you are using NUnit as your unit testing tool, for instance, you would simply prefix the NUnit command line with the NCover command.

NUnit command line:

NUnit-Console MyTests.dll

NUnit with NCover:

NCover.Console NUnit-Console MyTests.dll

It's easy enough to create code coverage data, but we also include the NCoverExplorer tool for browsing your source code with coverage analysis and HTML reports for publishing those metrics to your team.

Look at the NCover documentation for more information about how to use it.

We've had a great response to the release of NCover 2.0. Thanks to all of you who have tried out NCover 2.0 and gone on to purchase it. Keep it coming. The Gnoso team has a lot of fresh ideas and we're excited about bringing a whole new level of code quality tools to .NET developers!

We've had a few relatively minor issues reported in the past week and we've been fixing the problems and giving out patches to the people affected. We'll put another software update up on the site within a few days once the initial issues settle down. If you are aware of a problem that we may not have caught, please let us know so we can get it into this upcoming release. Again, thanks to those of you who have provided feedback. We're still listening...

The NCover project has entered a new era. If you've looked around the rest of the web site already, you've probably noticed the complete overhaul of our site. Since the NCover 1.5.8 release in April, we've been working on an upgraded version of NCover. We've made a large number of improvements in the product. Check the list of highlights on the "What's New" page and let us know what you think.

Over the past few years, NCover has become a tool that many development teams rely on daily for their continuous build processes. As such, we have gotten many requests for help and additional features. As a part-time project, there was no way I could give it the time it deserved. A short time ago, I began to realize that the only way to provide the kind of support that NCover needed was to create a sustainable business model around NCover. Over the past 6 months, I've been putting that business model together. I've hired a team of talented developers to help me maintain and support the product. We've partnered with (the sharp and tireless) Grant Drake to roll NCoverExplorer into the NCover project. The team has written sorely needed documentation for NCover and NCoverExplorer (and there's more to come). In short, NCover has needed to become an enterprise-class tool for .NET developers, and with 2.0 we've made that step.

In determining pricing for NCover, we've done our best to price it well within the range of comparable development tools in the market. NCover owes a debt of gratitude to the .NET developer community for putting up with a beta-quality product since the beginning of the project. So, in order to ease the transition of NCover from a free to not-free product we are taking the following steps.

1. Gnoso will provide a free NCover 2.0 Enterprise license plus a one year support subscription to everyone who made a donation of any size to the NCover project before today. (Yes, I have a list). If you are one of those people, please contact us to redeem your free license.


2. Gnoso will provide free NCover 2.0 Professional licenses to educational institutions for use in educating the next generation of developers. We believe that test-driven agile development processes need to be taught in computer science curricula and we want to support that effort in any way possible.


3. Gnoso will provide free NCover 2.0 Professional licenses to open source projects to support development of their projects. Many free/open source projects have embraced NCover by adding NCover integration to their tools or by using NCover in their testing and build processes. We'd like to see these teams continue to succeed. If you are an active developer on a .NET open source project, please contact us to get more information about working with the NCover team.

This is a monumental step for the NCover team and we know it's just the beginning. We are wide open to feedback from you, our customers. Please try out the new version and let us know how we are doing. The product is much improved, but I'm sure we didn't get everything exactly right. We'd love to know how NCover can work better for you.