Using NCover for .NET Code Coverage with Hudson

Code coverage can be quite a helpful tool in your project assessment toolbox, especially for giving you a view of untested areas of code. When I was tasked with generating coverage for our Outlook plug-in written in C#, I looked around and quickly settled on NCover as the tool to use. I chose NCover because it seemed like the most mature solution, already had integration with our test framework (MbUnit via Gallio), and could generate HTML reports and keep track of trends over time.

A few of the many NCover reports

A few of the many NCover reports

Integrating NCover with Continuous Integration

Generating XML or HTML reports was as simple as passing a few arguments to Gallio:

 C:\Program Files\Gallio\bin\Gallio.Echo.exe
    "C:\path\to\yourtest.dll"
    /runner:ncover3
    /runner-property:NCoverArguments="//html coveragedir //at ncover3.trend"

This will run the tests in yourtest.dll, output HTML reports in directory named “coveragedir”, and maintain trends over time in the file ncover3.trend. After doing this we had the first glimpse into the coverage of our unit tests. But just having these reports sitting in a folder on our build machine isn’t as useful as it could be. We wanted the HTML reports to be easily accessible from our continuous integration system, Hudson, so that anyone could easily view the coverage of the latest build. Hudson allows us to have all of our projects built and tested on schedules or on commit, and provides a central dashboard giving us a detailed view into many aspects of our projects, as well as notifies us as soon as something breaks.

Our Outlook dashboard in Hudson

Our Outlook dashboard in Hudson

NCover Plugin for Hudson

Unfortunately, there wasn’t an existing solution for integrating NCover and Hudson. However, as Hudson is open-source, has a friendly community, and a flourishing plug-in system, I decided to write an NCover plug-in for Hudson myself. After refreshing my Java skills and starting with the Javadoc Publisher plug-in as a base, it wasn’t long before I had a functional plug-in. We now have an NCover plug-in that can archive reports generated by NCover after a build and provide a link to them from the project or build page. If you are interested in more than one report, you can list as many as you like and each will show up as a tab on the report page. The plug-in has recently been released in the official Hudson repository and can be installed as easily as any other plug-in.

Installing the Plugin

To install it on your system:

  1. Visit your dashboard
  2. Click Manage Hudson -> Manage Plugins -> Available
  3. Scroll down and check “NCover
  4. Click the “Install” button at the bottom
  5. Restart Hudson to activate the plugin (it should offer to restart itself for you)
An example NCover plug-in configuration

An example NCover plug-in configuration

Configuring the Plugin

Once Hudson is restarted, go to the configuration page for the project which is using NCover. Scroll down to the “Post-build Actions” section and check “Publisher NCover HTML report”. The settings shown here will display two of the generated reports from the previously configured coverage directory.

After you save your NCover configuration and perform a build, you’ll now have a coverage report in Hudson, assuming your build is generating it!

Recommendations

Below is the main view of our Outlook plug-in, with the Code Coverage option now available at the top. The test results trend is courtesy of the Gallio plug-in, and I’d recommend using that as well if you are using Gallio. If you aren’t using Gallio, I’d recommend checking it out as a wrapper around your current testing framework as Gallio likely supports it.

Our Outlook plug-in in Hudson

Our Outlook plug-in in Hudson

View multiple NCover reports as tabs

View multiple NCover reports as tabs

Now whichever reports you specified in the NCover configuration will appear as tabs, allowing you to publish as many reports as you need. When you are done viewing the reports, simply click the “Back to Hudson” link at the top left. Now you’ve got .NET code coverage integrated with Hudson! Check out the NCover plug-in wiki page for more information and feel free to ask any questions or share any comments here. NCover also wrote about this plug-in on their blog, so check out their post too!

  • Digg
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Twitter
  • Google Bookmarks
  • DZone
  • HackerNews
  • LinkedIn
  • Reddit
  • Derik Whittaker

    I am trying to get this plugin to work, but it will never find the coverage html directory where my .html report is located.

    We have our coverage report stored in a folder path as such [WorkspaceRoot]/Results/TestResults/Coverage.

    I have tried the following
    **/Results/TestsResults/Coverage/
    Results/TestsResults/Coverage/
    Results/TestsResults/Coverage

    The exact error i am getting is
    Report: ‘CodeCoverage.html’
    ERROR: Specified NCover directory ‘Results/TestsResults/Coverage/’ does not exist.

    Do I need to put the FULL path name or what?

    Any help would be great. I love the fact you added this plugin, but I would love more to get it working. BTW, the hudson wiki has been down for 2 days (not your issue i know, but fyi)

  • Derik Whittaker

    I have seen that if i put the entire path (full from c:\ forward) it works, but this is NOT ideal

  • Mike Rooney

    Thanks Derik for checking out the plugin! The full path shouldn’t be necessary and a path relative to the workspace root works fine for me on Windows and UNIX. You say the tests are stored in a directory Results/TestResults/Coverage but say you are trying TestsResults, not TestResults, which looks like a typo. Is that the issue?

  • Derik Whittaker

    @Mike,

    Yes, that was a typo.

    I just removed the full path and tried both of the following
    Results\TestsResults\Coverage
    Results/TestsResults/Coverage
    **/Results/TestsResults/Coverage

    ERROR: Specified NCover directory ‘**/Results/TestsResults/Coverage’ does not exist.

    If i add back in the full path c:\…. it works

    Any more ideas?

  • Mike Rooney

    Could you post the full C:\ path so that I might better debug the issue? Thanks!

  • http://sholliday.spaces.live.com granadaCoder

    I think I found an issue.

    I think (not super sure, but I have a strong suspicion) .. it needs tested with a “custom workspace” option selected under project configuration (when using Subversion).

    For example:

    Source Code Management with (Subversion) (selected radio button)

    and then the _

    Configuration Item named:
    Local module directory (optional)
    with a value of _
    ./mytrunk

    I think that is disrupting the relative path-ness of the plug-in.

    Derik, were you using subversion with any options set?

  • http://www.genius.com Mike Rooney

    Thanks for the tip granadaCoder, I will have to check that out as I’ve never used that option before.

  • http://sholliday.spaces.live.com granadaCoder

    I just relooked. Under project configuration, there is also “Advanced Options” and “Custom Workspace”. Another wrench in relative pathness system.

    I did something like this.
    C:\Program Files\Hudson\HudsonHelper\CustomWorkspace\ProjectA\

    Again, I like to “flex” the system. I’m currently evaluation whether to use Hudson or CC.NET.

    ….

    PS I left a note over at https://hudson.dev.java.net/ (ncover plugin) about maybe providing some path information on a general exception.

    Thanks for this write up and your hard work!

  • Yrs One

    I have the same problem than Derik Whittaker.

    For example, if my path is :
    C:\Temp\HudsonData\jobs\Hello World\workspace\helloWorld\target\ncover.html

    What is the relative path ?
    Do you use %WORKSPACE% or ** ?

  • http://sholliday.spaces.live.com granadaCoder

    Here is another “hint”.

    No matter which value I set the “Coverage HTML Directory” ….I ~~always get the same javascript generated.

    MyFirstCoverageReport
    document.getElementById(“hudson_link”).href=”http://localhost:8080/”;

    (with emphasis on the value of the “value” attribute) (value=”MyFirstCoverageReport.html” in this case)

    First thing is that the “build” (when using a subversion pull-down) (when I don’t do any of the advanced options like I reported before) is that the file is located in:
    http://localhost:8080/job/FirstJob/ncover/trunk/MyFirstCoverageReport.html
    (emphasis on the extra “trunk” in the path)

    So even on default settings, I would expect to see the javascript output to be:

    MyFirstCoverageReport

    or something like that.

    Second, if I were to use a value for
    “Coverage HTML Directory”
    like “coveragedir”

    Then the javascript would be something like this:
    MyFirstCoverageReport

    ……..
    I think that highlights the issue.

    Unfortunately, I’m not a very good java developer, so getting the source code for the hpi plugin and fishing around there would be a painful exercise for me.

    The underlying issue is that when pulling down the build files via subversion, you’re getting (at least) a “trunk” thrown into the path. And if you settings values for “Local module directory (optional)” (Subversion setting under the project) and then also setting your “Coverage HTML Directory” property, you’re playing knick-knack-paddy-whack with the files and folders.

    I’d love to get NCover/Hudson working because I am evaluating hudson vs. cc.net. And the NCover reports not showing under Hudson is puttin’ a hurt on.

    I tried to see if the “pieces” were available at:
    localhost:8080/job/FirstJob/configure/api/xml/
    (where FirstJob is the name of my specific project)
    But (while there is lots of api/xml pages for getting at the metadata), there doesn’t seem to be one for the configure portion (/configure/api/xml/) :<

    Ok, I tried to be a part of the solution, not just part of the whining. Thanks!

    …………

    If you need a subversion test ground:
    http://beanstalkapp.com/
    allows you to create a personal and free subversion repository.

  • http://sholliday.spaces.live.com granadaCoder

    I found it!
    http://localhost:8080/job/FirstJob/config.xml
    (where FirstJob is the name of my project of course)

    Here are some of the snipplets from the xml…that could be used to piece together where the .html files are:

    ./mytrunk

    trunk
    MyFirstCoverageReport.html

    C:\Program Files\Hudson\HudsonHelper\CustomWorkspace\

  • http://sholliday.spaces.live.com granadaCoder

    G’aaacckkkk! The website is stripping my xml tags……so my post looks deficient.

    Here is some xml snipplets, with the formatting messed up so it will post:

    ./mytrunk

    trunk
    MyFirstCoverageReport.html

    C:\Program Files\Hudson\HudsonHelper\CustomWorkspace\

  • http://sholliday.spaces.live.com granadaCoder

    Geeze, here is another attempt to get around the formatting. Sorry for the multiple posts!

    Translation Key:
    ————————
    { = less than sign
    } = greater than sign

    {project}

    {scm class=”hudson.scm.SubversionSCM”}
    {locations}
    {hudson.scm.SubversionSCM_-ModuleLocation}
    {local}./mytrunk{/local}
    {/scm}

    {publishers}

    {ncover.NCoverArchiver}
    {coverageDir}trunk{/coverageDir}
    {indexFileName}MyFirstCoverageReport.html{/indexFileName}
    {/ncover.NCoverArchiver}

    {/publishers}

    {customWorkspace}
    C:\Program Files\Hudson\HudsonHelper\CustomWorkspace\
    {/customWorkspace}

    {/project}

  • http://sholliday.spaces.live.com granadaCoder

    Ok……I guess this is on the verge of being an abandoned project………No activity since October.
    Back to CruiseControl.NET I guess. The above issues I laid out are show stoppers……………….

  • Mike Rooney

    Hi granadaCoder, sorry you feel that way! I haven’t had any time to improve the plug-in since it works for our needs. I’d recommend filing the bug formally via the Issue Tracking link on the wiki page, and of course as Hudson is open-source, feel free to play around with it yourself.

    I’d also recommend trying the DocLinks plug-in which is a more generic plug-in for archiving reports and seeing if that works in your configuration. If so let me know and I’ll take a peek into how that is resolving paths and see if the NCover one can be improved in that way.