Coverage in the face of exception
Coverage in the face of exception
Hi there,
Thanks for NCover - I've just started using it today and it's
fantastic. One thing I've just noticed is that the closing brace of an
if statement is treated as "unreached" even if the point of the if
statement was to throw an exception. For example:
public void MyMethod(string myArg)
{
if (myArg == null)
{
throw new NullReferenceException();
} //this line is never "covered"
_myArg = myArg;
}
So the line with the comment is never considered covered because an
exception is always thrown when the if statement yields true.
Any thoughts on this?
Thanks again,
Kent Boogaart
Re: Coverage in the face of exception
This problem also occurs in this situation:
if (fooBar)
{
if (a == b)
{
return x;
}
else if (b == c)
{
return y;
}
} //this line is not "covered"
Kent
Re: Coverage in the face of exception
This is on my list of things to address.
Re: Coverage in the face of exception
peterwald wrote:
This is on my list of things to address.
Great, thanks :-)
Re: Coverage in the face of exception
I have a similar issue.
Quite often, I have to implement a finite state machine. Typical implementation is a case statement
select state
case state1
case state2
case else
throw new exception (....)
end select
The case else, should be there as a fall back if the state machine is wrong. It is never expected to be executed, just good practice. As such it will never show up on a code coverage. Also, you can't generate a test that will throw the exception.
Nick
Re: Coverage in the face of exception
...
The case else, should be there as a fall back if the state machine is wrong. It is never expected to be executed, just good practice. As such it will never show up on a code coverage. Also, you can't generate a test that will throw the exception.
Nick
Well, I guess you create the situation by using some mock object to stand in for your object state, but that aside, not having full coverrage is acceptable in this situation, as the code really ISNT inteded to be reached. Ever. (One of the good reasons you can have lower coverrage rate than 100% being shown as a succes in NCoverExplorer.)
Re: Coverage in the face of exception
Another example, probably easier to solve.
FxCop reports that you need to have a empty private constructor if you just have a class that consists of shared functions. Static in C#
It would be nice to do this
<NCoverIgnore()> _
private Sub New
End Sub
I can think of another example. If you have an obsolete function, you can ignore the fact that it is code covered or not.
I suspect that ignoring an entire method easier than ignoring part of a routine.
It is probably a pride thing of trying to get code coverage as high as possible.
On my original approach. Perhaps another way of solving the issue is to take the approach of FxCop. Allow the possibility of saying post analysis that a particular line is not relevant for code coverage.
ie. If we take this code again
we can remember the entire text of the method. We can mark the uncovered line(s) as 'ignore'. So long as the text of the method stays the same, we can ignore the lines. It is only if the method is changed, we don't know which lines to ignore.
I feel that NCover is the place to do this, so the results are available in both NCover and tools like NCoverExplorer
Nick
Re: Coverage in the face of exception
Nick,
You can do exactly your example already using NCover 1.5.4 - that is what Peter's "coverage exclusion attributes" feature is all about.
Even better, if you happen to define an attribute called "CoverageExcludeAttribute" without a namespace - e.g. I like to put it in an CommonAssemblyInfo.cs effectively so it is in each project...
internal class CoverageExcludeAttribute : System.Attribute {}
... then the recent TestDriven.Net 2.0.x versions will automatically specify on the commandline to NCover the necessary arguments, so when displayed in NCoverExplorer the code is automatically excluded - which is pretty cool.You can apply the [CoverageExclude] attribute defined on methods or classes.
The only downsides are that it requires the 1.5.x versions and hence .Net 2.0 to be installed on the machine, and that there are one or two known quirks with the current "beta"
implementation which you will find on this forum. Nothing that shouldnt be fixed eventually hopefully...
I must confess I am not a fan of the FxCop line# approach - it is just too damn brittle. Someone refactors the code/adds a new method etc and it all falls to bits.
Of course to round off your specific example if you are using .Net 2.0 in your app you can just mark the class static and dump the constructor anyways but your point is very valid for .Net 1.1.
This is an interesting problem to try to resolve that a workmate was debating with me this afternoon - developers effectively get "penalised" with their coverage statistics for studiously adding exception handlers and defensive code... like everyone else I look forward to the solution.
Re: Coverage in the face of exception
Did this closing brace issue ever get resolved?
I downloaded NCover today and started using it with TestDriven.Net in VS2005. But I'm discouraged because of the way it reports closing braces as code not covered.
For instance, even a very simple get accessor is reported as being only 66% covered, because it is not hitting the closing brace:
get { return employeeName; }
NCover counts three parts: the opening brace, the return statement, and the closing brace. It reports only 66% of the code is covered because it says the closing brace is not executed. This is really annoying.
I have the latest version of NCover: 1.5.5. What's the deal?
Re: Coverage in the face of exception
I get only two sequence points not three, and the coverage shows as 100% with NCover 1.5.5. With NCover 1.5.4 it has three sequence points, but they are all shown as 100% covered.
Try this below with a new class library in VS2005, right-click inside "MethodTest" and choose "Test with Coverage" (using the cool ad-hoc testing feature of TD.Net). This should show 100% coverage.
Also what version of TD.Net are you using, and have you most definitely got your NCover 1.5.5 installed at C:\Program Files\NCover?
namespace ClassLibrary1
{
public class Class1
{
public string TestProp
{
get { return "Test"; }
}
public void MethodTest()
{
string result = TestProp;
}
}
}
Re: Coverage in the face of exception
I'm using TestDrive.NET 2.0 personal edition - didn't want to pay for it until I was sure I wanted to use it. My NCover is in C:/Program Files/NCover. It's version 1.5.5 - I had version 1.5.4 on my machine and 1.5.5 won't install until you uninstall the old one. So I did that and rebooted.
Your test case didn't make any sense to me. This is what I did:
namespace ClassLibrary1
{
public class Class1
{
public string TestProp
{
get { return "Test"; }
}
}
[TestFixture]
public class TestClass1
{
[Test]
public void TestIt()
{
Class1 c = new Class1();
Assert.AreEqual("Test", c.TestProp);
}
}
}
That's using NUnit. Is there a problem using NUnit?
I'm certain there's something I'm missing here, but I can't find documentation on NCover anywhere.
Re: Coverage in the face of exception
I meant what version number of TestDriven.Net? Look in Help -> About in the VS.Net IDE. It will likely be 2.0.xxxx I guess to work in VS.Net 2005. Let me know what the xxxx is.
As far as the test case goes I posted - using the "ad hoc" test feature in TD.Net allows you to click inside any method and just run it regardless of whether it has been decorated with NUnit test attributes or not. So that code I posted simply allows you to "test" the same class the method is in - I wasn't interested in pass/fail from an NUnit perspective, just wanting the property to be hit and producing code coverage results. Great for quick and dirty execution of code when you want to try something.
Regardless of that I get excatly the same results with the class you posted - two sequence points and 100% coverage with NCover 1.5.5.
What version of NUnit do you have referenced - 2.2.8 for .Net 2.0 for instance?
If you like e-mail me your test project, the coverage.log file (in bin\debug), the coverage0.xml file produced (use File->Explore Coverage Folder from NCoverExplorer to find it easily) and the version information for both TD.Net and NUnit and I will see if I can replicate it. My e-mail address is in Help ->Send Feedback in NCoverExplorer. What o/s are you running as well?
Re: Coverage in the face of exception
I can't get any sort of ad hoc feature to work. Maybe it's a limitation of the trial version of TestDriven.
The version is: TestDriven.NET 2.0.1761 Personal
NUnit was 2.2.6 for .Net 2.0. I downloaded the 2.2.8 version and installed - still no change.
I'm sending you the logfile and xml though - there's a lot of errors in the log.
Re: Coverage in the face of exception
Chris,
Ok, the problem is the combination of versions of NCover, NCoverExplorer, TestDriven.Net you are using. I drafted a blog entry about this but never got around to publishing it. It's to do with incorrect information that NCover 1.5.5 publishes into the coverage file based on the way TestDriven.Net reflects on assemblies.
To cut a long story short, both Jamie and I made changes to our applications to prevent this problem so the solutions are to either:
(1) Download the latest beta of NCoverExplorer, or
(2) Download the latest version of TestDriven.Net (later than 2.0.1761), or
(3) Use NCover 1.5.4
You will find the issue disappears when you have done this. This applies to anyone who upgraded to NCover 1.5.5 - please upgrade NCoverExplorer or TD.Net. I am not sure if Jamie has officially released a later build than 2.0.1761 at this point on his website and he is offline at the time I write this but if not please use the NCoverExplorer upgrade path instead.
As for the ad hoc testing - no that should not be disabled. You should be able to click inside any method in your application and choose "run tests". If the method has parameters, they will be passed some values like empty string, zero etc.
Re: Coverage in the face of exception
Thanks Grant.
I checked out the TestDriven.Net site again just to make sure I wasn't hallucinating, and I can't find a version later than 2.0.1761. That appears to be the latest version of the Personal edition.
I'll try the beta of NCoverExplorer, and if that doesn't work I'll go back to NCover 1.5.4.
Edit: Replaced the NCoverExplorer - it works. Thanks Grant!
Re: Coverage in the face of exception
Hey Peter,
Here's a poke to make sure this is still on your list of things to address. I'm cool with exclusing TestAttribute to work around the [ExpectedException()] case that this comes up in, but for methods that are just a 'throw new NotImplementedException()', my coverage is still skewed.
I'd be happy to donate $50 to see a fix implemented for this, if an automated test is also added in the process.
Thanks! :)
Re: Weird results with classes using generics
Interesting indeed. If you can come up with a repeatable case let me know. Also, make sure that both machines are running the exact same routines, otherwise the coverage would differ. Also, if the code is running under extreme load in a multi-threaded environment, you may have some coverage counts that are lower than normal. The coverage percentages would still be solid though.
Re: Weird results with classes using generics
Hi
I forgot to mention that the weird results occur when the coverage is run from cruise control dot net.
Re: Weird results with classes using generics
Sounds like I need to look into CruiseControl. There are a lot of people reporting problems with it and NCover.
Re: Weird results with classes using generics
Hi
I've taken a further look at the issue and only one of my classes in not producing the same code coverage results when run locally versus when run from cruise control dot net.
How can I help in correcting this issue? Do you need a log file? Do you need the source code of the class that is causing the problem?
Thanks
Re: Weird results with classes using generics
I have fixed a bug in the upcoming release 1.5.2 related to registry settings and the user not having enough permissions. This may fix the problem you are seeing.
Re: NCover repository
A subversion repository would be nice ;-)
Re: NCover repository
Hi Ramon,
The source code is still available for browsing here:
http://www.svn-hosting.com/trac/NCover/browser/
There was a link on the old NCover start page, but I think Peter has forgotten to add it to the new site. Or I just haven't found it yet, I managed to locate the repository through Google.
- Kim
Re: <ncoverexplorer> task's <exclusions>: what about <inclusions
Hi mate,
Can you give a more specific example where this would be useful?
You can of course use wildcards on your exclusions and I had found this to normally be sufficient with how we name our assemblies/namespaces. Normally we are either excluding test assemblies (e.g. *.Test.*) or specific layers (*.Presentation.*) or third party/in house libraries that get caught in the crush since the pdbs exist in the folder. The latter wouldn't be an issue if we had a way with TestDriven.Net of specifying the equivalent of //a for NCover but that's another issue.
If I understand you correctly you are saying that rather than doing what we do above, we could instead say inclusions of "MyApp.Business.*" and "MyApp.Data.*" (for instance), and by default anything else is excluded?
Would it be acceptable to be an either/or thing? i.e.
- you either specify "exclusions" like people do currently (where the default is to include anything and then apply the exclusion list)
- or you specify "inclusions" (where the default is to exclude everything but this list)
I guess what I would want to try to avoid is the problems of combining inclusions and exclusions which can cause no end of confusion. (e.g. if I can include "MyApp.*" and exclude "*.Tests.*" then what happens to "MyApp.Tests.dll"?).
Depending on how people name their assemblies/namespaces this may actually make it more difficult and verbose (particularly for large projects).
There is another problem this would introduce that won't be "fixable" until project support is added to NCoverExplorer. That problem is for people who use the GUI against multiple projects. Either your inclusions list has to list all the projects you work with, or you have to keep changing it every time you load a different project's coverage file up.
It's an interesting suggestion - I'm happy to discuss it further but guess I need to see a concrete example of where this would be of help.
Re: <ncoverexplorer> task's <exclusions>: what about <inclusions
Hi kiwidude,
I actually did had in mind an either/or notion of how this can be accomplished; I can't see where someone will want (or need) to combine exclusions and inclusions.
Here's my case:
I have two kind of projects in my team: the first (let's call it "server") is a core application contributing assemblies to all the other (let them be "clients") with ECI binary dependencies over CCNet. The thing is that I can't distinguish "server" assemblies from "client" assemblies because untill now the naming convention was poor; example:
this is an assembly of the "server" app: companyname.server.service1.dll
and this one from the one of "client" apps: companyname.server.client.service1.dll
thus, using this wildcards like "companyname.server.*" will exclude the "client" assemblies from the "client" report as well... There is an understanding for this problem and a new assembly was named after this pattern: companyname.client.service1.dll which will solve the problem only partially (because the misnamed are vital for future releases as it looks now).
Thanks,
-M
Re: <ncoverexplorer> task's <exclusions>: what about <inclusions
Sounds like you are dropped in it by some pretty poor naming decisions - I understand your problem now.
I guess another possible solution to this would be to allow the use of regular expressions instead? It might be a bit more flexible. So you could say "server but not followed by client" etc in your expression.
Re: <ncoverexplorer> task's <exclusions>: what about <inclusions
regexp is a great solution! how come I didn't thought about it :-)
Re: <ncoverexplorer> task's <exclusions>: what about <inclusions
I've just pushed up build 1.3.5.1861 which should give you regular expression pattern support for coverage exclusions. Give it a go and let me know how you get on. If you use the NAnt/MSBuild tasks you will need the latest version of the Extras.zip as well.
http://ncover.org/SITE/forums/thread/664.aspx
Re: <ncoverexplorer> task's <exclusions>: what about <inclusions
It works great!
I've checked it with NCoverExplorer GUI & with NAnt task (<ncoverexplorer>) only. Also the documentation is fantastic, it took me less than 15 minitues to change my nant script, test it over and over untill I was ready to push it to my ccnet project...
Thanks again,
~M
Re: <ncoverexplorer> task's <exclusions>: what about <inclusions
Thank you!
I also appreciate using regexp to filter out exluded namespaces.
Re: Out of memory error
Mark,
Thanks for the bug report. I have just pushed a new build up which should hopefully prevent the null reference exception. The out of memory errors I too have seen very occasionally - perhaps if I get time this weekend I will do some memory profiling to see if anything obvious jumps out.
http://www.kiwidude.com/dotnet/NCoverExplorer.1.3.5b.zip
Re: Out of memory error
The link http://www.kiwidude.com/dotnet/NCoverExplorer.1.3.5b.zip ; leads to a 404 error.
The original link posted with the announcement of the 1.3.5 beta is http://www.kiwidude.com/dotnet/NCoverExplorer-1.3.5b.zip, which works and has the latest version.
I'm just posting this to help out anyone else reading this thread.
- Mark
Re: Out of memory error
Thanks Mark - sorry just typed it off top of the head instead of copy/pasting. Curious as to whether anyone else is experiencing out of memory errors - my colleagues all said no at work today, although perhaps they don't keep it open long enough and refreshing it as much as you indicated you had...
Re: Out of memory error
An update on this for you - I finally had a chance to look into this a little myself tonight and there is something nasty going on with the TreeView control (closing coverage files does not appear to be releasing much if any memory). This is now my #1 priority to resolve for the 1.3.5 release - hopefully get this resolved this weekend.
Re: Out of memory error
I have just pushed up a fix for this which should hopefully resolve one of the leak issues (opening/closing source code tabs). Download the latest 1.3.5 beta release (build 1813 onwards) as described in this thread:
http://ncover.org/SITE/forums/2/615/ShowThread.aspx
Let me know if you are still experiencing problems.
Re: NCoverExplorer.Console.exe Problems
Hi Farek,
The ModuleClassFunctionSummary is a function coverage report. It is not a method level report. Hence the main difference you will see in the output between those two report types is a difference in the percentages where /r:4 is sequence point coverage % (per class per namespace per assembly) and /r:5 is function coverage % (per class per namespace per assembly). There is one other difference you missed in the reportTitle attribute of <coverageReport> - this tells the xsl stylesheet to give a slightly different look to the output.
There is a feature request placed for method level reporting - it is on the todo list for a future release but no specific date as yet. There are a lot of reasons why it is not there currently (slowing down the CC.Net build server pages with all the extra data it requires is one of them, lack of raw data available from the coverage.xml to identify overloads is another). My personal belief is that if you really want to go to method level coverage you want to see the lines of code not being hit - which means using NCoverExplorer interactively (or HTML reporting displaying the source code which is also on the todo list).
As for your other problems with the paths - I will take a look later to try to replicate and will push a new build up with any changes. Thanks for providing all the detail.
Re: NCoverExplorer.Console.exe Problems
I have just pushed a new build up (1.3.5.1850) which fixes the two relative path bugs. See http://ncover.org/SITE/forums/thread/641.aspx for the download details and let me know if you experience any problems from this.
Thanks for reporting the issues.
Re: NCoverExplorer.Console.exe Problems
Thank you for the quick feedback :)