Forgot Password?Register
Log In Download
29
Jul '09
by Daniel Waldschmidt

Kathy's Wrong!

One of our NCover fans emailed me outraged at a recent article in Visual Studio Magazine where Kathy Dollard slammed code coverage while answering a question about unit testing.

In the last paragraph of answering the question Kathy makes the case that:

Code coverage is a terrible metric -- no better than counting lines of code as a measure of productivity. Code coverage can never determine whether the system is well tested, because it can't consider whether boundary conditions (such as null, negative and zero values) and other likely failure points were considered. The only thing coverage indicates is what code has not been tested at all.

Kathy, you are wrong!

There are very few (if arguably any) objective measurements for building high-quality code. Dynamically measuring code quality is really the only one that provides accurate "health" of code. John Robbins was our guest at a recent webinar where he talked about the "real life" significant savings of applying code coverage. He made a strong case that enterprise problems demand trustworthy analysis -- more specifically the analysis derived from running code coverage.

And it's key to note that code coverage isn't just limited to unit testing. A significant number of NCover's customers use code coverage analysis in a code base where unit tests do NOT and will NOT ever exist...

Symbol level unit test coverage (as referenced by Kathy) of a code base is severely inadequate as the only metric on which to base code health. Our community has been evangelizing this message for years.

It's silly -- like suggesting that a band-aid is the right fix for heart palpitations just because you have lots of band-aids available in your medicine cabinet. Getting the pulse of code health is a sophisticated, multi-tier strategy that involves testing code coverage throughout the entire build/test/deploy process.

Many developers get the impression that their code coverage choices are limited to unit test coverage simply because the basic code coverage tools in Microsoft's Team Foundation Server are only rudimentary.

That's not reality!

You need thorough code coverage on every part of your solution, and there are tools that can make this a seamless process. I can't speak first-hand about other code coverage tools, but here's a quick run-down on NCover's holistic approach to code coverage:

  • Unit testing -- Each developer is responsible for their part of the solution and NCover provides multi-metric feedback in comparison to complexity
  • Manual testing -- NCover watches the functional and integration testing done by your QA/Testing team (locally and off-shore)
  • Security testing -- Ncover provides audit reporting (if you can "hack" to much of the app with your tests than we make a big deal about that to you... and blow up the build)
  • Data-driven testing -- NCover provides individual coverage analysis along with overall trending of testing process
  • QA tool testing coverage -- NCover connects to the .NET testing application of choice (popular choices include Quick Test Pro and TestComplete...)and let's you know what pieces of the puzzle you are missing
  • User acceptance testing (UAT) -- NCover allows us to showcase how much of an app was touched in high-level "button pushing" testing
  • Build testing -- NCover drives build reporting back through management with options for failing builds based on satisfaction thresholds

Now, layer that coverage data and see what your overall code coverage is. You might find that:

  • 41% unit test coverage PLUS
  • 77% manual test coverage PLUS
  • 0% security test coverage PLUS
  • 62% QA test tool coverage PLUS
  • 65% build test coverage GETS YOU

pretty darn close to an absolutely delightful-bug-free solution.

What are your thoughts?

View Comments

10
Jul '09
by Peter Waldschmidt

Code Quality

As the creator of NCover I often get asked what my inspiration was and what the importance of code quality and coverage really is. Well, it’s really very simple:

Customers expect more.

They want software publishers to make their software available for instant download over the Internet, pushing updates frequently. They expect software to install quickly and easily and work right the first time out of the box. Software that upgrades itself is even better. All of these things contribute to your customer's expectations for a quality product. So how can we consistently achieve quality?

Quality is not something you can add to a product; it must be built into the foundation.

It has been said that if we regularly pay attention to something it will improve, while what we ignore will languish. Software quality is achieved by not only paying attention to it, but also by understanding what quality really is.

So, what do I mean by quality?

There are qualitative measures of quality like:

  • Does the software solve the customer's problem? Does it do what it was designed to do?
  • Does the software meet all of the specifications that were defined when starting the project?

These aspects are sometimes measurable, but may just result in arguing over the definition of the problem your software hopes to solve. The measures that I want to focus on right now are the quantitative measures.

  • How many bugs have my customers found in my code since it was released?
  • Have our support requests increased or decreased since the new release? What is the ratio of support calls to users?
  • Has the complexity of my code improved or decreased over time? How understandable is it?
  • How many automated tests do I have? are they passing?
  • What code is tested by my automated tests? More importantly, what code is NOT tested by those tests?
  • Is my code getting slower or faster over time?

These are pieces of the development process that cannot be ignored.

A decade ago, source code version control software was the only real tool requirement that a software shop had aside from a compiler and maybe a text editor. If you were doing version control, then you were writing software right. Now, we've reached the point where, if you are not doing version control then you are either lazy or ignorant and there are even higher expectations like continuous integration, automated testing, stress testing, code coverage and static analysis.

My primary message is this:

Put in place measurements at key places in your development process and know what those numbers mean. Watch them and improve them. Measurement is the first step on the way to building a software development process that produces high quality products.

View Comments