Getting Started with Jazz Code Coverage

Summary

The Code Coverage component provides an easy and efficient way to measure the test coverage while your projects evolve. A tight integration into the Jazz workbench together with an easy-to-use reporting mechanism help any developer to quickly get test coverage results without spending too much time with the code coverage tooling. With the optional build step to produce test coverage results during a Jazz build, the Jazz Code Coverage tooling is even easier to use, and test coverage is measured automatically in Jazz builds without any user intervention nor configuration on the Jazz client.

By Code Coverage Component Team
Jazz Development
IBM Rational
December, 2007

---

Introduction

The Jazz Code Coverage component integrates code coverage measuring into the team’s build process and provides an easy-to-use interface to get code coverage results, both on the server (while producing builds of your code) and on the client (while launching a Java application). This document is divided into three parts: Part one explains how to set up a Jazz build to produce test coverage results, part two discusses how test coverage results can be obtained from locally running JUnit tests, and part three explains how code coverage results are best browsed in the Jazz user interface.

Prerequisites

We strongly recommend to first read the following documents:

• The Jazz Tutorial
• Jazz Platform – Client Setup Guide
• Getting Started with setting up Jazz Builds

This document assumes that a Jazz server has been set up and you have access to it. Furthermore, it is assumed that you have successfully set up a Jazz build using an example build script from the above-mentioned article.

To be able to view coverage reports on the Jazz client, you need to install the JazzCodeCoverage-IXXXXXXXX-XXXX.zip download available from the Jazz Project Incubators list.

Part 1: Configuring a Jazz build to produce test coverage results

Getting test coverage results from a Jazz build is straight-forward. However, depending on the environment on the build server and already existing build scripts, there is no generic way to augment a build script to produce Jazz code coverage results. This is the reason why the Jazz Code Coverage tooling only offers a set of customizable ANT tasks that can be used to enhance existing ANT build scripts to perform test coverage measurements.

The process of computing test coverage results in a Jazz build happens in five steps:

• compiling: the source code is compiled to a set of output directories.


• instrumenting: the output from the compile step (Java .class files or JAR files) is instrumented with probes to collect coverage information when executed later. This step can be controlled by coverage filters, a mechanism to define the scope when collecting coverage information or generating coverage reports. This produces a set of meta-data files which are kept for coverage report generation.


• testing: the build drives the tests based on the instrumented code, while collecting coverage information. The build scripts passed various additional arguments to the virtual machine running the tests, such as the boot class path entry to the coverage engine runtime and paths arguments defining the locations where to store coverage information after the tests terminate.


• reporting: a coverage report is generated based on the collected meta-data and coverage data.


• publishing: the created coverage report is published to the Jazz build result.

For each one of the steps outlined above, Jazz offers a corresponding ANT task if the functionality is not already provided by ANT itself.

The Jazz Code Coverage tooling comes with the following set of ANT tasks to implement the instrumenting and reporting step:

  <project>  	<taskdef name="instr" classname="com.vladium.emma.instr.instrTask" />  	<taskdef name="report" classname="com.vladium.emma.report.reportTask" />  	<taskdef name="coverage" classname="com.ibm.team.coverage.common.coverageTask" />  	<taskdef name="analysis" classname="com.ibm.team.coverage.common.analysisTask" />  </project>  

These ANT tasks are provided by the following JARs:

• emma.jar
• emma_ant.jar
• com.ibm.team.coverage.common_0.6.0.IXXXXXXXX-XXXX.jar

Please refer to the Documentation section for further information about those ANT tasks. The file CoverageToolkitTaskDefs.xml in the Jazz build toolkit directory provides ready-to-use task definitions to be included into your build scripts.

Moreover, the Jazz Code Coverage tooling contributes an example build script which is based on a simple compile and test scenario, similar to the compile-and-test example build script in the examples subdirectory of the Jazz build toolkit directory. You can use the compile-and-test-coverage example build script as the basis to create your own Jazz build script, or use parts of the script to integrate test coverage into your existing ANT build script.

The compile-and-test-coverage example build script is basically an adoption of the compile-and-test build script with appropriate calls to the ANT tasks provided by the Jazz Code Coverage tooling. It features four new targets, init, instrumentCode, createReports and publishReports, plus some additional properties to control various aspects of the code coverage report generation.

Existing build scripts may use those targets and insert them at the appropriate locations, according to the five steps necessary to run tests using code coverage measurements. Please make sure that init is called before other code coverage ANT targets are invoked.

Part 2: Launching Applications to measure code coverage

Launching a Java application such as a JUnit test is easy. To find out how well your code is covered, you just need to follow three steps:

Instrument Projects

• Go to the project properties of your project (Alt+Enter), browse to the Coverage tab, and enable static code instrumentation.

  

• Alternatively, if you would like to instrument multiple projects at once, you might also use the Run > Coverage > Instrument Projects... action.

Run Applications

Run one or more Java applications to drive the code you would like to measure. The Jazz Coverage tooling not only works with JUnit tests, it works with all Java-based launches supported by Eclipse as well!

Obtain Results

There are two ways how to obtain coverage results from a launched application: The label decorations in Java-based views such as the Package Explorer and the Outline view, or a customized coverage report based on an analysis over several Java projects and application launches.

• Observe the coverage decorations which are shown in the Package Explorer or the Outline view after every application launch:

  

• Perform a coverage analysis to produce a customized coverage report: Choose Run > Coverage > Run Analysis... from the workbench menu and select the Java projects and application launches you would like to have included in your report:

  

• Press Run and browse the created coverage report:

  

  On the second page of the coverage report viewer, you will find some overall statistics about the coverage report:

  

The coverage report viewer offers rich sorting and grouping functionality. Java elements can be grouped by Jazz component (if available), Java package or Java type, and each column can be sorted to quickly spot the candidates for creating new test cases or improving existing ones. If you open the associated source files, Jazz will ask you to update any coverage information which is out-of-date.

Part 3: Browsing code coverage results

Once you have created a test coverage report or you are browsing a Jazz build result, you can start exploring the coverage report and see which parts of your code base need improvements with respect to test coverage:

Use the actions Show in Java Editor and Show in Package Explorer to navigate to the source code in your local workspace. If you are currently browsing a build result with code coverage information, Jazz will ask if the coverage information has not been downloaded or the locally available coverage information is out-of-date. In this case, you can let Jazz automatically update your local Eclipse workspace.

Downloading Results

If you have a Jazz build result with associated coverage information, you may want to start digging into your code to see how the test coverage of some classes could be improved. Fortunately, there is no need to run all tests again locally to get this information for the local Eclipse workspace.

• In the Coverage tab of a build result, choose the action Download to Workspace.

This action “projects” the coverage information of the build result onto your local Eclipse workspace, giving you the feeling as if all tests of the build have been run locally.

• Open the Package Explorer and browse the Java projects which have been compiled for the current build result. Note how the Java elements are annotated with coverage information from the build result.

• Open a source file from the Package Explorer which is annotated with coverage information. Note how the Java Editor displays a ruler on the left side with line coverage information, which is constantly updated while you edit the source file.

The presentation of the coverage information in the Package Explorer and the Java Editor may be configured on the preference page Java > Coverage.

• Once you are done exploring the coverage of your code in the local workspace, you may choose the action Run > Coverage > Clean Decorations... to get rid of all coverage information in your local Eclipse workspace.

Coverage Filters

Coverage filters are a means to limit the information overhead normally produced by a full coverage report covering several projects. They provide an easy way to specify which Java elements should participate in code coverage via inclusion/exclusion filters.

To specify coverage filters on a Java project, go to the project preferences of your project (Alt+Enter), browse to the Coverage tab, and edit the coverage filters.

On the Coverage Filter dialog, you can specify the scope and type of the filter. Coverage filters may be specified in the popular syntax known from the Open Type (Ctrl+Shirt+T) command.

The syntax follows the familiar *,?-wildcard class name patterns and has the following format:

• [package.]type, the optional package name in dot notation, followed by the type-qualified name of the type
• * matches any string
• ? matches any character

Active Launch

The concept of an active launch is used in the Jazz user interface to specify the data source for coverage information such as the Package Explorer decorations and the line-level coverage decorations in the Java editor.

Every time a launch involving an instrumented project has terminated, the Jazz coverage tooling records the code coverage measured by that particular launch, and the decorations in Package Explorer and Java editor are updated accordingly.

There are however situations in the code-test-fix cycle where you would like to have this information stable, meaning that it does not change every time a launch has terminated. This is the reason for an active launch, which can be configured to always represent the most recently terminated one, or a particular launch from the launch history which does not necessarily have to be the most recent one.

Keep the checkbox Use most recently launched as active launch checked if you would like to always have the most recent coverage information displayed. Otherwise, uncheck the checkbox, select the desired launch, and press Set active.

Examples

Build script examples with optional test coverage support can be found in the /jazz/buildsystem/buildtoolkit/examples folder of TeamConcert-IXXXXXXXX-XXXX.zip. A build script that enables coverage for the JUnit Sample project is shown here:

<!-- This example demonstrates a very simple ant build that uses the jazz build toolkit and jazz scm. --> <project name="sample" default="all"> <tstamp> <format property="timestamp" pattern="yyyyMMdd-HHmm" /> </tstamp> <!-- Typically, a "build" user is created and used here. --> <property name="userId" value="build" /> <property name="password" value="build" /> <property name="datadir" value="data" /> <property name="workingDir" value="${basedir}/.."/> <property name="build.output" value="${workingDir}/bin" /> <property name="compileLog" value="${java.io.tmpdir}/compilelog.xml"/> <!-- Various paths used while generating coverage reports. --> <property name="coverageDir" value="${basedir}/coverage" /> <property name="buildToolkit" value="${basedir}/../../../../buildtoolkit" /> <property name="buildId" value="${timestamp}" /> <property name="buildDefinitionId" value="JUnit" /> <property name= "summaryFile" value= "${coverageDir}/coverage.summary" /> <property name= "analysisFile" value= "${coverageDir}/coverage.analysis" /> <property name="zippedReportDownload" value="Coverage-${buildDefinitionId}.zip" /> <property name="zippedReportFile" value="${coverageDir}/${zippedReportDownload}"/> <property name="zippedAnalysisFile" value="${coverageDir}/Analysis-${buildDefinitionId}.zip"/> <property name="reportDir" value="${coverageDir}/report"/> <property name="reportIndex" value="${reportDir}/index.html"/> <!--Paths of the Emma meta-data and coverage data files. --> <property name="metaFile" value= "${coverageDir}/coverage.em" /> <property name="traceFile" value= "${coverageDir}/coverage.ec" /> <!-- VM arguments to pass to the VM executing the JUnit tests. --> <property name="outVMArgument" value="-Demma.coverage.out.file=${traceFile}" /> <property name="bootVMArgument" value="-Xbootclasspath/a:${buildToolkit}/emma.jar"/> <target name="all" depends=" fetchCodeAndBuild, invokeTests"> </target> <target name="fetchCodeAndBuild"> <startbuildactivity label="publishing links" autocomplete="true" buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <linkpublisher buildresultuuid="${buildResultUUID}" verbose="true" label="Jazz.net" url="https://jazz.net" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <startbuildactivity label="compiling" activityidproperty="compileActivityId" autocomplete="true" buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <property name="build.debuglevel" value="lines,vars,source"/> <property name="build.compiler" value="org.eclipse.jdt.core.JDTCompilerAdapter"/> <mkdir dir="${build.output}"/> <startbuildactivity label="compiling the source" parentactivityid="${compileActivityId}" autocomplete="true" buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <javac destdir="${build.output}" failonerror="false" debug="on" includes="**/*.java, *.java" srcdir="${workingDir}"> <compilerarg line="-log ${compileLog}"/> </javac> <startbuildactivity label="publishing the compile results" parentactivityid="${compileActivityId}" autocomplete="true" buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <jdtcompilelogpublisher buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" filepath="${compileLog}" /> <startbuildactivity label="publishing the compiled binaries" parentactivityid="${compileActivityId}" autocomplete="true" buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <jar jarfile="${workingDir}/junit-${buildLabel}.jar"> <fileset dir="${build.output}"> <include name="**/*.class"/> <exclude name="**/*samples/*.class"/> </fileset> </jar> <artifactfilepublisher buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" verbose="true" filepath="${workingDir}/junit-${buildLabel}.jar" label="JUnit JAR" /> </target> <target name="invokeTests"> <!-- Instrument code to get coverage results while running tests --> <antcall target="instrumentCode" /> <startbuildactivity label="testing" activityidproperty="junitActivityId" autocomplete="true" buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <mkdir dir="${workingDir}/test.results"/> <startbuildactivity label="running the tests" parentactivityid="${junitActivityId}" autocomplete="true" buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <!-- Just a little delay to make the nest activities look better in the build result editor --> <sleep seconds="5"/> <junit showoutput="true" fork="yes" failureproperty="testFailed"> <jvmarg value="${bootVMArgument}"/> <jvmarg value="${outVMArgument}"/> <formatter type="xml" /> <test name="junit.samples.AllTests" todir="${workingDir}/test.results" outfile="junit.samples.AllTests.results" /> <classpath> <pathelement path="${workingDir}/bin"/> </classpath> </junit> <startbuildactivity label="publishing the test results" parentactivityid="${junitActivityId}" autocomplete="true" buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <junitresultpublisher buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" filepath="${workingDir}/test.results/junit.samples.AllTests.results.xml" verbose="true" failonerror="false" /> <!-- Create coverage reports after the tests finished --> <antcall target="createReports" /> </target> <target name="instrumentCode"> <startbuildactivity label="Instrumenting Code" activityidproperty="compileActivityId" autocomplete="true" buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <path id="buildtoolkit.lib" > <fileset dir="${buildToolkit}" /> </path> <!-- Instrument all class files in the destination directory and update the Emma meta-data. --> <coverage> <instr mode="overwrite" merge="yes" metadatafile="${metaFile}" verbosity="verbose"> <!-- Exclude any JDK classes. --> <filter excludes="java.*, javax.*" /> <instrpath> <!-- Include everything in the project output folder. --> <dirset dir="${build.output}" includes="**/*"/> </instrpath> </instr> </coverage> </target> <target name="createReports"> <mkdir dir="${reportDir}" /> <startbuildactivity label="Creating Coverage Report" activityidproperty="compileActivityId" autocomplete="true" buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" /> <path id="buildtoolkit.lib" > <fileset dir="${buildToolkit}" /> </path> <dirset id="project.dirs" dir="${workingDir}" includes="*" /> <!-- Create the requested coverage reports. --> <coverage> <!-- Create a HTML report from the collected meta-data and coverage data. --> <report units="instr" depth="method" columns="name, class, method, block, line" sort="+name,+class,+method,+block" metrics="method:70,block:80,line:80,class:100" sourcepath="${workingDir}" verbosity="verbose"> <infileset dir="${coverageDir}" includes="*.em, *.ec"/> <html outfile="${reportIndex}"/> </report> <!-- Create a Jazz coverage analysis and coverage summary. --> <analysis projectsref="project.dirs" flags="0" sourcepath="${workingDir}" outfile="${analysisFile}" summaryfile="${summaryFile}" verbosity="verbose"> <infileset dir="${coverageDir}" includes="*.em, *.ec"/> </analysis> </coverage> <zip destfile="${zippedReportFile}" basedir="${reportDir}" /> <!-- Zip the files. --> <zip destfile="${zippedAnalysisFile}"> <fileset file="${analysisFile}" /> </zip> <!-- Publish the created reports. --> <antcall target="publishReports" /> </target> <target name="publishReports"> <!-- Create a publish destination directory. This directory will hold the published report for the particular build. --> <property name="publishDestination" value="${basedir}/${buildDefinitionId}/${buildId}" /> <mkdir dir="${publishDestination}" /> <!-- Copy the HTML coverage report to the publish destination. --> <copy todir="${publishDestination}" file="${zippedReportFile}" verbose="true" overwrite="true" failonerror="false" /> <!-- Publish the HTML coverage report. This report will be available in the downloads tab of the Jazz build result editor. --> <artifactlinkpublisher buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" verbose="true" label="Coverage Report for ${buildId}" url="${publishDestination}/${zippedReportDownload}" failonerror="false"/> <!-- Publish the coverage analysis report. This report will be shown in a separate tab in the Jazz build result editor. --> <filepublisher buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" contributiontypeid="com.ibm.team.build.coverage" verbose="true" filepath="${zippedAnalysisFile}" label="Coverage Report" failonerror="false" /> <!-- Publish the coverage summary. This information will be used in the Jazz build result editor for the coverage overview. This step is optional and can be omitted. --> <filepublisher buildresultuuid="${buildResultUUID}" repositoryaddress="${repositoryAddress}" userid="${userId}" password="${password}" contributiontypeid="com.ibm.team.build.summary" verbose="true" filepath="${summaryFile}" label="Coverage Summary" failonerror="false" /> </target> <taskdef name="artifactFilePublisher" classname="com.ibm.team.build.ant.task.ArtifactFilePublisherTask" /> <taskdef name="artifactLinkPublisher" classname="com.ibm.team.build.ant.task.ArtifactLinkPublisherTask" /> <taskdef name="buildResultPublisher" classname="com.ibm.team.build.ant.task.BuildResultPublisherTask" /> <taskdef name="filePublisher" classname="com.ibm.team.build.ant.task.FilePublisherTask" /> <taskdef name="jdtCompileLogPublisher" classname="com.ibm.team.build.ant.task.JdtCompileLogPublisherTask" /> <taskdef name="junitLogPublisher" classname="com.ibm.team.build.ant.task.JUnitLogPublisherTask" /> <taskdef name="junitResultPublisher" classname="com.ibm.team.build.ant.task.JUnitLogPublisherTask"/> <taskdef name="linkPublisher" classname="com.ibm.team.build.ant.task.LinkPublisherTask" /> <taskdef name="logPublisher" classname="com.ibm.team.build.ant.task.LogPublisherTask" /> <taskdef name="completeTeamBuild" classname="com.ibm.team.build.ant.task.CompleteBuildTask" /> <taskdef name="startTeamBuild" classname="com.ibm.team.build.ant.task.StartBuildTask" /> <taskdef name="getNextTeamBuildRequest" classname="com.ibm.team.build.ant.task.GetNextBuildRequestTask" /> <taskdef name="requestTeamBuild" classname="com.ibm.team.build.ant.task.RequestBuildTask" /> <taskdef name="teamAccept" classname="com.ibm.team.build.ant.task.TeamAcceptTask"> </taskdef> <taskdef name="teamFetch" classname="com.ibm.team.build.ant.task.TeamFetchTask"> </taskdef> <taskdef name="workItemPublisher" classname="com.ibm.team.build.ant.task.WorkItemPublisherTask"> </taskdef> <taskdef name="startBuildActivity" classname="com.ibm.team.build.ant.task.StartBuildActivityTask" /> <taskdef name="completeBuildActivity" classname="com.ibm.team.build.ant.task.CompleteBuildActivityTask" /> <!-- Task definitions that required for coverage tooling --> <taskdef name="instr" classname="com.vladium.emma.instr.instrTask" /> <taskdef name="report" classname="com.vladium.emma.report.reportTask" /> <taskdef name="coverage" classname="com.ibm.team.coverage.common.coverageTask" /> <taskdef name="analysis" classname="com.ibm.team.coverage.common.analysisTask" /> </project>

Documentation

Detailed documentation of the Jazz Ant tasks can be found at /jazz/buildsystem/buildtoolkit/task-javadoc/index.html in TeamConcert-IXXXXXXXX-XXXX.zip.