Kathy's Wrong!
Kathy's Wrong!
One of our NCover fans emailed me outraged at a recent article in Visual Studio Magazine where Kathy Dollard slammed code coverage while answering a question about unit testing.
In the last paragraph of answering the question Kathy makes the case that:
Code coverage is a terrible metric -- no better than counting lines of code as a measure of productivity. Code coverage can never determine whether the system is well tested, because it can't consider whether boundary conditions (such as null, negative and zero values) and other likely failure points were considered. The only thing coverage indicates is what code has not been tested at all.
Kathy, you are wrong!
There are very few (if arguably any) objective measurements for building high-quality code. Dynamically measuring code quality is really the only one that provides accurate "health" of code. John Robbins was our guest at a recent webinar where he talked about the "real life" significant savings of applying code coverage. He made a strong case that enterprise problems demand trustworthy analysis -- more specifically the analysis derived from running code coverage.
And it's key to note that code coverage isn't just limited to unit testing. A significant number of NCover's customers use code coverage analysis in a code base where unit tests do NOT and will NOT ever exist...
Symbol level unit test coverage (as referenced by Kathy) of a code base is severely inadequate as the only metric on which to base code health. Our community has been evangelizing this message for years.
It's silly -- like suggesting that a band-aid is the right fix for heart palpitations just because you have lots of band-aids available in your medicine cabinet. Getting the pulse of code health is a sophisticated, multi-tier strategy that involves testing code coverage throughout the entire build/test/deploy process.
Many developers get the impression that their code coverage choices are limited to unit test coverage simply because the [basic code coverage tools in Microsoft's Team Foundation Server] (http://docs.ncover.com/white-papers#extend-vs "basic code coverage tools in Microsoft's Team Foundation Server") are only rudimentary.
That's not reality!
You need thorough code coverage on every part of your solution, and there are tools that can make this a seamless process. I can't speak first-hand about other code coverage tools, but here's a quick run-down on [NCover's holistic approach] (http://www.ncover.com/products#ncover-fits "http://www.ncover.com/products#ncover-fits") to code coverage:
- Unit testing -- Each developer is responsible for their part of the solution and NCover [provides multi-metric feedback] (http://docs.ncover.com/best-practices/code-quality-metrics "provides multi-metric feedback") in comparison to complexity
- Manual testing -- NCover watches the functional and integration testing done by your QA/Testing team (locally and off-shore)
- Security testing -- Ncover provides audit reporting (if you can "hack" to much of the app with your tests than we make a big deal about that to you... and blow up the build)
- Data-driven testing -- NCover provides individual coverage analysis along [with overall trending] (http://docs.ncover.com/how-to/tracking-code-coverage-trends/ "with overall trends") of testing process
- QA tool testing coverage -- NCover connects to the .NET testing application of choice (popular [choices include Quick Test Pro] (http://docs.ncover.com/how-to/running-ncover-with-your-unit-testing-framework/quick-test-pro "choices include Quick Test Pro") and TestComplete...)and let's you know what pieces of the puzzle you are missing
- User acceptance testing (UAT) -- NCover allows us to showcase how much of an app was touched in high-level "button pushing" testing
- Build testing -- NCover drives build reporting back through management with options for failing builds based on satisfaction thresholds
Now, layer that coverage data and see what your overall code coverage is. You might find that:
- 41% unit test coverage PLUS
- 77% manual test coverage PLUS
- 0% security test coverage PLUS
- 62% QA test tool coverage PLUS
- 65% build test coverage GETS YOU
pretty darn close to an absolutely delightful-bug-free solution.
What are your thoughts?
RE: Kathy's Wrong!
One thing that I find particularly interesting with high coverage ratio is when portion of code covered are also relying on contracts. Contracts are correctness verification 'inside' the code itself. The more the code covered contains contracts, the less risky you have uncovered bugs.
I wrote also about Why code coverage is a good thing Anyway! (Anyway means even if you are not testing everything)