Comprehensive test plan reporting with RPE and RRDG

Marc van Lint, IBM The Netherlands
Last updated: 1 January 2014
Build basis: Rational Quality Manager 4.0

Rational provide many options to report on information stored in the Jazz and other databases. This article describes the creation and setup of a Rational Publishing Engine template to print a Test Report. The template is shared which can be used in the RPE Launcher and RQM/RRDG. This template can be considered an additional RQM/RRDG sample template.

To fully understand this article intermediate knowledge of RQM and RPE is required. Other articles available on internet will give an introduction to these products. Check the websites Jazz.net, IBM developerWorks, http://rpeactual.com, http://business.vanlint5.nl

A quick look on a template result can be found here RQM-RRDG-TP-V29.pdf

Contents:

1. Are we ready to ship?
2. Product Introduction
3. Best Practices
4. Comprehensive Test Plan
5. Templates and Results
6. RRDG Limitations
7. Conclusion

“Are we ready to ship?”

This question was raised in one of the first marketing movies on RQM. It puzzled me, because it is a very difficult question to answer. There is no simple answer. When asking around to various test managers you get different answers often depending on the industry or project they are involved in. Common to the answers is that every test manager want to get an overview over requirements, open defects, test results and their relations.

This template generates a document which gives a comprehensive overview on the status. It basically combines many standard RQM reports, the feedback from the test managers and the leading edge of document generation. The report shows:

1. All involved requirements and if they are successfully tested
2. All tests and test results, per environment and time line if applicable
3. Statistics on defects
4. Graphs on test execution

This template will give a test manager a better insight in the status of his project. That should help him to write is advice if the product is ready to ship!

Product introduction

This section briefly describes the various software components which are used in this articles.

Rational Quality Manager

IBM Rational Quality Manager is a collaborative hub for business-driven software and systems quality across virtually any platform and type of testing. This software helps teams share information seamlessly, use automation to accelerate project schedules and report on metrics for informed release decisions.
More on RQM can be found here

Rational Publishing Engine

IBM Rational Publishing Engine automates document generation from Rational solutions and select third-party tools. You can use Rational Publishing Engine to automate the generation of documents for ad hoc use, formal reviews, contractual obligations or regulatory compliance.
More on RPE can be found here

Rational Reporting for Document Generation (RRDG)

RRDG is the ‘engine’ to generate the reports based on a template and data from resources. This engine is initiated from the RQM GUI. Templates can be defined by the RPE application.
More on RQM can be found here

Best Practices

While working on these and other templates the following working methods proved to be valueble.

1. Run your template frequently while developing.
2. Use one text file to store various examples URLs and snippets of JavaScript.
3. Make versioned backups of your templates (MyTemplate-V01.dta, MyTemplate-V02.dta, MyTemplate-V03.dta). When you are happy with a template you can delete old versions.
4. Use a debug variable to conditionally activate prints/logic. When true the debug information is printed.

   

   Figure: By adding condition you can easily switch them on/off leveraging the debug variable.

   Make sure you select the debug value in the list.

   

   Figure: Setting the condition based on the debug variable.

   Personally I give de text printed enabled by the debug switch in a different color/font, e.g. red (FF0000).

   

   Figure: An example of the formatting of debug information in the output.

5. Print status and values in the Console and rpe.log. These statement must be placed in a script block (can be used in combination with the debug variable) which is executed.

     _sessionLogger.info( SYS_TemplateName + "(TP_Section Test Case Execution Records)");  

   
   You can leverage inbuild RPE logging using the following sample code:

     // Utilizing JavaScript   java.lang.System.out.println( "This will be shown in console" );    // _sessionLogger methods  _sessionLogger.info( "This will be shown in the console as well as in the rpe.log file");  _sessionLogger.debug( "This will only be shown in the rpe.log file");    // logging docspec information  _sessionLogger.info( "RPE version number: " + _sessionInfo.engineVersion);  _sessionLogger.info( "RPE build number: " + _sessionInfo.buildNumber);    // this property is only populated at runtime. Editing the script and closing the script window  // will show "" as the result in Studio  _sessionLogger.info( "Template path (requires docgen): " + _sessionInfo.getTemplateProperty( "path",""));    									

   It’s very informative when you have to generate a document which takes a long time to process. This way you can print out a progress of the execution.
6. Standardize your naming of variables and data sources (see below)
   • Use same names for variables as in RQMAPI documentation
   • Some variables are according to ‘your standards’, e.g.
     • TPId
     • TPSummary
     • TPQueryUrl
     • TPLock
   • Use CaMeLcAsE for readability.

   This will improve readability of the templates you create, it will improve the listing of the variables in the views, and typing errors will surface quickly.

   

   Figure: Standardizing variable names will assist in using the right ones.

7. Inherit the userid/password from the very first/root data source defined (e.g. the test plan). This way you only have to define once the userid/password in the execution template. See also chapter “Credentials Inheritance”.
8. Give each container a name, it highly improves readability in the template

   

   Figure: The setting of the name of the container.

   The naming makes the RPE Outline very readable. This way you can easily identify sections in the template. It proofs to be very powerfull in combination with the Zoom Editor.

   

   Figure: The container naming visible in the outline.

   This naming can be applied also to other elements, a great help when searching for the right location.

   

   Figure: Naming of elements helps in identification..

9. Use the Zoom Editor and selecting in the Outline.
   Some structures need deep nested elements. A big help is using the zoom editor and selections the elements of interest in the Outline (assisted by the enhanced naming).

Comprehensive Test Plan

Specific characteristics of the template:

1. Many Reports in One Template
2. 35+ Variables to Customize
3. Support of Many Sections
4. Included: (Requirements) Traceability
5. Included: Test Case Design
6. Included: Statistics on Defects
7. Links to Artifacts in GUI
8. Graphics
9. Correct naming conventions of Data Sources
10. Credentials Inheritance
11. Metadata added to the Test Plan Data Source

Many reports in one template

The power of the template is the way it can be customized automatically or easily. By setting the various variables one can have a different behavior and layout (more on this in the next paragraph). A major switch is the SW_TP_ShowActiveSectionsOnly variable.

1. Manual inclusion of sections

     SW_TP_ShowActiveSectionsOnly = false;  

   The template defines which sections are to be shown in the template. This way you quickly see which sections are not populated with data despite the visibility of the section in the Test Plan.

2. Automatic inclusion of sections

     SW_TP_ShowActiveSectionsOnly = true;  

   The information gathered and filtered is based on the visibilty of relevant sections in the Test Plan.

Figure: The setting of the various switches (SW_TP_ prefix) to steer the template.

With the SW_TP_UseVisible set to true, the template will include only those sections which are visible in the test plan. Based on the visibility of the Test Environments and Test Schedule sections in the test plan, RPE generates reports with the inclusion of the appropriate test results and information. Not applicable results are excluded.

Figure: The explanation at the end of the report on which results are included.

35+ Variables to Customize

There are 35+ variables to control the behavior of the document generation. These include:

1. Switching on/off of sections
2. Make use of Traceability links
3. Colors
4. Show/Hide Test Case Design
5. Show/Hide Closed Work Items
6. Enabling graphics

When the SW_TP_ShowActiveSectionsOnly is set to true, the sections shown in the Test Plan are also influencing the generation. For example hiding Test Environments will influence the selection of data and layout of the test results and graphs.

Figure: The setting of the various switches (SW_TP_ prefix) and colors (Color_ prefix) to steer the template.

Support of Many Sections

When a Test Plan is created in the GUI, you probably will use the default test plan template. This will give a certain selection and order of the sections available. This RPE Template will also process the sections which will be revealed when using a different Test Plan Template (e-signature, test bed etc).

Figure: Most sections are processed in this template.

Note: Several sections are not processed, for example Risk Assessment and Development Plan Links. These sections are not yet included in this sample template.

Included: (Requirements) Traceability

There are several locations where traceability to requirement is reported. To name a few:

• Test to Test
  For example: Test Plan to Test Cases
• Requirements to Test
• Test to Requirements
• Related Work Items
• Related Defects

Figure: A sample section where traceability is visible in the report.

The traceability between test artifacts are used in this template. An example is the relation between a test plan and test case. When the template is use from the RPE Launcher the traceability to the requirements and defects/work items are automatically taken into account.

Some of the links are easily retrieved from the received data. A simple following of the href will do. Sometimes other queries are to be used to gather the information you want to process. An example is the relation to the Work Items.

Included: Test Case Design

The template is enhanced in such a way that the test case design is included in the Test Case sections. This is essential when the Test Case contains important information, like BDD information to steer Cucumber.

Figure: When switched on the Tes Case Design is included in the Test Case section.

Included: Statistics on Defects

The list of related defects and tasks is included. When you want you can hide the closed work items. A table with statistics on the open defects is also provided.

Figure: A summary statistics on the defects.

Links to Artifacts in GUI

Where possible every ID is linked to the actual source artifact. This gives the reader the ability to view details and update the artifact in one click.

Figure: The link of an ID.

Graphics

RPE has no graphics engine, but we can leverage external graphics engines via the JavaScript options. In this template leverages the Google graphics to generate and include basic graphics. The used API is depreciated by Google, but for this example it works great. There are several other JavaScript Graphics engines which can be leveraged.

Figure: A sample graphics showing pass/fails of related tests to a requirement.

The basic is simple to gather information from the various sources, feed that to the engine and include the image in the template.

Figure: A sample graphics showing trends per environment based on the executed tests.

Correct Naming Conventions of Data Sources

Applying a naming convention to variables and datasources helps in easy identification and (typing) error detection. It improves the readability of the template and the includes JavaScript.

1. Use underscore to hide data source in RRDG GUI (more)
2. When non RQM, precede with RRC or CCM.
3. If it’s a listing use name followed by an s, e.g. “_Things”
4. If it’s for details use name followed by Details, e.g. “_Thing Details”

Figure: The naming of the Data Sources.

Credentials Inheritance

Not to fill in many times the same credentials over and over, you can inherit these from another data source. This way you have to specify only once the credentials.

Figure: Inheritance of the credentials of the Test Plan Data Source.

Note: This can be used for test data sources and other CLM data sources when the RPE Launcher is used. When run as RRDG template, this will only work for test data sources, not for CLM data sources.

Metadata added tot the Test Plan Data Source

The template needs a seed to start from. This can be a complete database or a specific (test) artifact. Because we want to a have a Test Plan report, it’s obvious that we identify the report and use it as a starter.

The test plan identification can be defined in the RPE Specification when using the RPE Launcher.

Figure: The identification of the Test Plan in the RPE Specification.

The actual URI is the following line (all on one line):

  https://business.vanlint5.nl:9443/qm/service/com.ibm.rqm.integration.service.IIntegrationService/          resources/My%20Reporting%20%28Quality%20Management%29/testplan/urn:com.ibm.rqm:testplan:5  

The number at the end is equal to the ID of the Test Plan in the RQM GUI. Make note that the CLM Project name is encoded using the %20, %28 and %29 characters. These URLs can be send to RQM using the POSTER plug-in in Firefox. (RQM and Poster)

When the template is started from the RQM GUI, leveraging the RRDG engine it needs some additional information to setup the starting GUI. We already hide the various data sources by using the underscore in the name. The test plan identification needs some more information, meta data.

Figure: The additional Metadata to the Test Plan Data Source.

The Request URL contains the following URI (all on one line):

  ${public}/service/com.ibm.rqm.integration.service.IIntegrationService/resources/${projectAreaUUID}/                        testplan?fields=feed/entry/content/testplan[title='${userInput}']/*&wildcard=*  

This will result in a very basic GUI to identify the Test Plan. That will generate the URI for the RPE Template.

Note: Make note that for successful processing of traceability you need additional definitions in the RRDG engine.

RRDG Limitations

There are some noteworthy differences between launching from RPE Launcher and the RQM/RRDG GUI. Although it’s not the target of this article, some differences are listed here. This is not a complete list and will change over time with new releases of CLM/RPE software.

1. The RPE Launcher is capable to extract data from RQM, RRC/DOORS NG/DOORS, RTC and other sources. The RRDG engine needs additional customization to access non-RQM data.
   Related Jazz work item: WI 252251.

   In line with the Work Item the following definitions are added:

   

   Figure: Additional data sources are required!

   

   Figure: Additional metadata to the data sources! These are the pre-populated servers which are used.

   

   Figure: The additional data source definitions results in two additional fields which probably should be ignored by the user.

2. When RRDG generates a report, it’s done with the HTML layout. A font size is set with an integer in px, about 15 is a good start. When you generate a PDF layout the these font-sizes are roughly 50% larger than the comparable HTML fonts.
   This behavior affects the use of RRDG.
3. In RRDG there is no option, known to me, to define style files, like in the RPE launcher.
4. The RQM RRDG GUI is very limited in defining parameters. Currently only string variables are available. With some help to identify a test artifact. There are no options to select from a list, validate for integer values or set Boolean etc. The is no option to add some descriptive text to guide the user.
   Related Jazz work item: WI 101732.
5. In RRDG you can’t supply associated images to be included in the template. A work around is to store the images on a webserver and reference these.
   Related Jazz work item: WI 293163.
6. The PDF exporter handles colspan of table cells wrongly.

Thorough testing is needed before putting a RPE template into production using RRDG.

The Template and Result

Due to the differences between the RPE Launcher and RRDG environment, two templates are provided here for each environment

Template Source	Template Result
RQM-RRDG-TP-V29.dta	RQM-RRDG-TP – V29.pdf

How to use these templates in both environments is well described in the following article RQM RRDG.

The actual images, files and results may differ due to new releases of Rational software, different settings or different versions of templates.

Known Limitations of this Template

It’s an example, it’s intended to trigger idea’s and jump start any custom reporting leveraging this techniques. The example has several limitations like:

1. Naming of time slot should start with a year-month combination. This is needed also for sorting.
2. All artifacts must reside in a CLM environment. Doors or CQ will not work with this template. It’s possible to include these and other OSLC data sources.
3. The requirements, work items and defects should be on the same host. It starts a query and check’s if there is a relation to the test plan.
4. TC weight can be listed easily, reporting is currently done on Test Execution Record counting.

Conclusion

To complement RQM with RPE report generation is a logical one. RQM provides various default, customizable reports. RPE gives the ability to have full control on what’s reported and how. It’s very well fitted for document generation.
The RRDG gives great opportunities to have your own test-reports run from the RQM GUI.

---

For more information

• RQM Homepage, RQM RRDG
• RPE Homepage, RPE Forum
• The Auhor’s own website with it’s focus on Rational Quality. New versions of this template will appear here.
• rpeactual.com – A great source to follow when actively involved in RPE usage.

---

About the author

Marc van Lint started in 1986 at IBM as a intern. He was active for 18 years in the Product Lifecyle Management branch of IBM. In 2005 he continued his career at IBM Rational. His prime focus was to achief Quality in the software delivery process at the various customers. He presents his visions to customers and help them to implement their improvements. He contributes frequently to the community by publishing on the The Rational Tester Blog, maintaining his own website focusing on Rational Quality Tools. He’s co-author of the IBM Press book “Software Test Engineering with IBM Rational Functional Tester – The Definitive Resource”.
This is his third article on Jazz.net. Other articles are “Automated Test Deployment with the Rational solution for Collaborative Lifecycle Management – Part 1” and “Part 2”.
He can be contacted at marc_van_lint@nl.ibm.com.

---

Copyright © 2014 IBM Corporation