Jazz Library A look inside LQE and Report Builder
Author name

clm,elm,jazz-reporting-service

Mon, 09 Apr 2018
5 min read
0

A look inside LQE and Report Builder

Kevin Cornell, Persistent Systems
Kathryn Fryer, IBM
Last updated: February 2018
Build basis: IBM Collaborative Lifecycle Management Solution (CLM) 6.0.3 or later, IBM IoT Continuous Engineering (CE) Solution 6.0.3 or later

This article describes how the Lifecycle Query Engine (LQE) data source and Report Builder (RB) tool process metadata from applications and data sources to support reporting in the IBM Collaborative Lifecycle Management (CLM) and IBM IoT Continuous Engineering (CE) solutions. It complements the article Integrating external data sources with LQE and Report Builder, and is intended to help developers who are integrating their data sources understand how that data is surfaced to reporting users. RB administrators and report authors will also gain a better understanding of what they see in the RB UI and generated report queries.

The LQE data source uses tracked resource set (TRS) feeds contributed by lifecycle applications to create and maintain an index of link-accessible resources. The index is synchronized periodically with each data source, ensuring that data is current and providing near-real-time results. In the CLM/CE offerings, Report Builder provides the user interface to build interactive reports against the LQE data sources, generating SPARQL queries to execute those reports. For configuration-enabled projects and versioned data, LQE is the required data source for reporting. Rational Engineering Lifecycle Manager (RELM) also uses LQE as a data source, but is not addressed in this article.

LQE provides two endpoints, which appear in Report Builder as two data sources:

  • LQE (sometimes referred to as “all-data”). This data source is available when you install and register LQE.  When you select this data source in RB, you see data from all projects that do not support configuration management, and any that are “compatible”. You also select this data source to report on configurations and components (which are not versioned resources themselves).
  • LQE scoped by a configuration. This data source is available only when configuration management has been enabled for at least one project. An administrator must then initialize this endpoint in the RB Admin UI by clicking Connect to Data Sources.  When you select this data source in RB, you see data from all projects that have enabled configuration management, and any that are “compatible”. To run a report against this data source, you must specify a configuration context.

LQE also has two special partitions: “shapes” for resource shapes and merged shapes, and “history” for historical metrics. These are relevant only in terms of understanding how LQE processes data; they are not surfaced to the user or administrator and you don’t interact with them directly.

To populate the user interface and generate queries, RB first builds metamodels for each data source, including the projects, artifact types (based on resource shapes), and properties.  (Note that RB also provides the same capabilities for building interactive reports using the data warehouse; for more information on those capabilities, see the IBM Knowledge Center.)

The topics in this section describe how LQE and RB process and store the information that applications publish, generate metamodels for reporting, and show the metadata to the report author in the user interface to generate SPARQL queries and reports, including:

The appendix Understanding merged resource shapes provides more technical details on how LQE and RB merge metadata for resource shapes to facilitate cross-project reporting, which is primarily of interest to developers contributing external data to LQE and report creators building complex custom queries. The companion article Integrating external data sources with LQE and Report Builder provides details on the metadata required to support all of these operations. 

This article references concepts and terms related to OSLC, RDF, and TRS fundamentals. If you require more background, see the additional resources and background materials listed; many other introductory materials exist on the internet.

Access control for reporting data

LQE and RB use project access control settings to manage user access to the reporting data. Each project area defines an Access Control list, often based on project membership. LQE automatically synchronizes permission to resources based on that access control list. RB enforces the access control in the report editor and when a user runs a report.

The application TRS feeds define access context resources for each project area; all the resources in the project area (process resources, resource shapes, instances) reference the appropriate access context for the project area they belong to, as well as the project area itself. Applications that don’t use project areas can either simulate project areas and provide access contexts, or make theire resources accessible to all authenticated JRS users for reporting. Access context resources are described in more detail in the article Integrating external data sources with LQE and Report Builder.

LQE stores the access context and associated users, and lists them as User Groups in the LQE administration UI. Administrators can modify groups and create new ones. Administrators can also grant additional permissions to groups or individual users to access data in project areas where they are not included in the access control list.

RB can access all the data in LQE, using the credentials of the JRS functional user. When RB generates the reporting metamodels, each type and property has a list of the projects that published them. In the user interface, RB filters the list of projects in the Limit Scope section based on the user’s access permissions, and filters the types and properties based on those projects and any selections in Limit Scope.

Note: If an external application or data source makes their resources accessible to all authenticated JRS users, selecting one or more projects in the Limit Scope section or at run time will hide those resource types, properties, and instances, because they do not match any project area.

For more details on how RB enforces access control by project area, see the later section Scoping based on project permissions.

The reporting metamodel

For each reporting data source, RB generates a memory-based metamodel hierarchy: a collection of closely related meta-types, based on the type URIs. In RDF terminology, an RB metamodel corresponds to an OWL ontology. RB loads the metamodels when you launch the report editor, using the cached metadata to populate the user interface (for example, the entries and domain groupings in the Choose an Artifact section), and to generate the queries for the report.

Every data source has multiple associated metamodels. Each metamodel usually corresponds to a specific domain (rm, qm, and so on), and contains one or more meta-types, each with meta-properties. 

The meta-types in a metamodel are based on RDF types (of type rdfs:Class) from static vocabularies, and on OSLC resource shapes. Although the RDF syntax does not allow types to have super-types, resource shapes can inherit and further describe properties from an RDF type. In the metamodel, those resource shape types and merged shape types will have super-type references that correspond to its described RDF types.

Some changes in application data require a metamodel refresh before the reporting user can see them. RB automatically refreshes the data source metamodels based on a defined schedule; the section Refreshing the metamodels explains how administrators can change that schedule, and also manually refresh the metamodels if needed. 

How Report Builder generates the metamodel

Report Builder builds the reporting metamodel primarily by querying the metadata in LQE. For these queries, RB uses the credentials of the JRS functional user; this user is established when you set up the LQE server, and has access to all indexed data in LQE.

Before querying the metadata, RB tests its connection to the LQE data source, validating the JRS functional user can run a simple query and that instance data exists for reporting. If the test connection fails, RB shows the user an error indicating the metamodel was not loaded.

After validating the connection, RB creates metamodels for:

  • Each RDF ontology (resources of type owl: Ontology).
  • Each group of merged resource shape types in the “shapes” partition. RB combines results for versioned and non-versioned shapes. By default, RB loads only the merged shapes; if it can’t find the “shapes” partition, it creates metamodels for each group of project-specific resource shape types in the default partition instead. For more details on merged shapes, see Appendix: Understanding merged shapes
  • Each group of resource shapes for the metrics types in the “history” partition, if one exists.

All shape types with the same URI prefix are included in the same metamodel. For each metamodel, RB defines the associated meta-types based on the published triples for each OSLC or merged shape type, and metadata on the service providers and project areas for each resource shape.

Finally, RB queries the metadata for the properties for each type, including enumeration values and any properties that define external URIs (declaring owl:sameAs in the metadata) or reversible relationships (declaring jrs:inverseLabel). RB combines the results to define the meta-properties for each meta-type.

Note: When configuration management is enabled, RB generates a single set of metamodels that are shared by both the LQE and “LQE scoped by a configuration” data sources. The metadata itself is not typically versioned, except for versioned resource shapes which are handled by shape merging.

Refreshing the metamodel

Report Builder generates and caches the reporting metamodel, refreshing it on a defined schedule. The default schedule is twice daily at 6:00am and 6:00pm. As applications publish modified metadata to LQE, some changes affect the metamodel and will not be visible to report authors and users until the metamodel has been refreshed. Changes that require a metamodel refresh include:

  • Adding a new project
  • Creating a new type
  • Creating a new attribute or link type
  • Modifying the name of a type (because type merging is based on the type name)
  • Specifying an external URI for an existing enumeration attribute value
  • Specifying an external URI for an existing attribute or link type
  • Changing the name of an attribute or a link type
  • Modifying the target type of an attribute or link type (this type of change is not recommended)

Administrators can change the RB refresh schedule by modifying the metamodel.autorefresh.time and metamodel.autorefresh.repeat.inminutes settings in the server/conf/rs/app.properties file.

Administrators can also manually refresh the data source from the Report Builder administration UI (https://server:port/rs/endpoint) by editing the data source and clicking Refresh.

Setting timeout values for metamodel queries

LQE provides settings to limit how long a query can run and the number of results it can return. Metamodel queries have their own timeout and result size parameters, because they can take longer to run and return a much larger result set than report queries. By default, metamodel queries time out after 10 minutes (600 seconds), and can return unlimited results (value of -1).

You can modify the timeout setting for metamodel queries by setting the lqe.read.timeout property in the server/conf/rs/app.properties file. Modify these values with care to ensure RB can still generate the complete metamodel.

For report queries, the default result set size is 3000 results and timeout limit is 60 seconds. You can change these values in the LQE Admin UI or the lqe.releaseconfig.properties file. See the IBM Knowledge Center and Jazz.net LQE Advanced Properties article for more details on configuring LQE.

Interacting with the metamodel to build and run reports

In addition to generating the metamodel for reporting, Report Builder also provides the user interface for building the reports, surfacing the metamodel to the report author in a usable way.

Report authors don’t see the metamodel generation, although they might see a message when they select a data source indicating that RB is loading the metamodel. If an error occurs during loading, the report author will not be able to continue with the report using that data source. The metamodel does inform how types and properties appear in the user interface, as described in the topics that follow.

Scoping based on project permissions

LQE and RB use access control settings from each project area to determine user access to the reporting data. As described in Access control for reporting data, users automatically have access to data for the projects where they are included in the access control list, based on the access context lists defined in the project area and TRS feeds. LQE administrators can grant additional project area access to individuals or groups.

When a user builds or runs a report, RB queries LQE for the list of projects that the user can access. RB displays the list of available projects in the Limit Scope section of the report editor, and in the runtime filters.

Report Builder also scopes the list of projects based on whether they are enabled for configuration management, and the selected data source:

  • LQE:  includes all projects that are not enabled for configuration management and do not have versioned artifacts, and all projects that are “compatible” with both enabled and non-enabled projects (meaning they do not have versioned data, but can link or integrate with projects that do). That includes projects from RTC and GCM applications.
  • LQE scoped by a configuration:  includes all projects that are enabled for configuration management, and all “compatible” projects.

Note: When using LQE, you can choose to List all projects when reporting on configurations themselves. Because configurations aren’t versioned, use the LQE data source (not LQE scoped by a configuration) to report on them. If you wanted to count all versions of an artifact across configurations, you would choose to include all those projects. However, you should not report on any details for those artifacts; because the report is not scoped to a configuration, the results for a query on the artifact could yield conflicting results.

In the RB metamodel, the types and properties indicate the project areas that published them. RB filters the types and properties in the UI to show only those published by the project areas the user can access. If the user selects projects in Limit Scope, the types and properties are further filtered by that selection.

Applications can choose to publish resources that are available to all JRS authenticated users. Those types and properties would be visible to all report users. However, selecting any projects to limit scope hides those resources, since they are not associated with any project area.

When a user runs a report, the results are also filtered based on the user’s project access permissions, and any projects selected to limit scope. Different users can run the same report, see a different list of projects, and get different results based on their access permissions.

For projects with configuration management enabled, the list of configurations shown in the runtime filter is also scoped by any project selections in either the report definition or the runtime filter itself, and by any components specified in the Choose a Component runtime filter. See Scoping configurations at run time for more details on how RB determines the list of available configurations to select at run time.

Understanding type hierarchies

In the Choose an Artifact section of the report editor, RB shows the types for each domain in a hierarchy with the most generic types at the top and more specific types below. The base RDF types are the most generic; shape-based types are more specific, and where they describe an RDF type, appear below that RDF type.

For example, under Requirements Management, one top-level artifact is the generic OSLC Requirement; any requirement types defined in the RM application appear under that generic Requirement type:

Modules and requirement collection types are similarly arranged under the generic OSLC Requirement Collection type.

The generic OSLC artifact types for Quality Management are hidden because each has only a single merged shape type from RQM (QM Test Case, QM Test Plan, and so on).

For Change and Configuration Management, the base RDF type is the OSLC Change Request type. The RTC Work Item type and various shape types (Defect, Story, Task, and so on) extend the OSLC Change Request. Because the Work Item is more generic, RB nests the other shape types under Work Item as though they were derived from the Work Item type. RB also hides the OSLC Change Request type because it effectively has only one derived type (Work Item). If another application publishes resource shapes to LQE that describe or extend the OSLC Change Request, RB will display the OSLC Change Request type with the derived types shown nested below, and as siblings to Change Request. For example, if IBM ClearQuest published ClearQuest Defect types to LQE, the expanded type hierarchy might look like:

Change Request 
   ClearQuest Defect
   Work Item 
      Defect 
      Story 
      Task 
ClearQuest Defect
Work Item 
   Defect 
   Story 
   Task

Understanding artifact properties

Each meta-type in a metamodel has one or more meta-properties. When the user selects an artifact type in the report editor, RB displays the list of properties defined directly by the selected type, and those defined by its super-types. If both the selected type and one of its super-types define the same property (with the same predicate URI), the definition in the derived type overrides that of the super-type.

There are two kinds of properties:

  1. Attributes have a target of an XSD type (Boolean, date, decimal, integer, string) or the foaf:Person type. If the target type is a resource, but the shape definition allows only specific resources or URIs to be assigned as values, the property is an enumeration and considered an attribute. Report authors can use attribute properties in conditions and add them as columns to a report.
  2. Relationships have a target type of another resource type. Relationship properties define links between instance resources; report authors can use them only in a traceability path. When you select a relationship property in a traceability path, RB displays a list of all possible target types based on the defined target type and any derived types. Users can select a more generic type, for example to report across multiple projects, or a more specific derived type, for example to include a specific property not found on the generic type.

Note: The properties displayed are always filtered based on the project access permissions of the report author or user. If the author selects projects in the Limit Scope section of the UI, those project selections further filter the types and properties displayed. See Scoping based on project permissions for more details on how project permissions affect what RB shows in the UI.

Reporting on relationships

As described above, to include artifact relationships (links to other resources) in a report, use the Traceability setion in Report Builder. For a given relationship between linked resources, it’s possible that only one of those resource types asserts the relationship in its metadata. Requiring the report author to know which types assert the relationships is not practical; users might want to trace the relationship between the types in either or both directions.

If applications publish appropriate metadata, RB can establish “reversible relationships” so users can report in either direction regardless of which instance resource asserts the relationship. The Jazz.net article Integrating external data sources with LQE and Report Builder describes the metadata in more detail.

There are two possible scenarios for relationships:

  1. Only one of the types asserts the relationship in its published metadata. For example, a test plan has an oslc_qm:usesTestCase property, but the test case metadata has no inverse relationship property to the test plan.
  2. Both types define the relationship in their metadata; either or both types can assert the relationship in the metadata. This includes OSLC-defined pairs of link properties, such as oslc_cm:testedByTestCase and oslc_qm:testsChangeRequest.

Note: When you enable configuration management, linking is directional; only the source instance resource publishes the relationship triple. However, the metadata for both types might still define the relationship properties, like the OSLC link relationships defined by the OSLC specification.

For either scenario, RB generates metadata in the reporting metamodel to create a reversible relationship. Reversible relationships are completely transparent to the user: in the report editor, users can select either artifact type as a starting point, and trace the relationship in either direction.

RB generates different report queries depending whether one or both resources assert the relationship in the metadata. If only one type can assert it, regardless of which type the user selects as a starting point, RB queries only the asserting type for the relationship property. If either type can assert the relationship, RB queries the relationship on both types and does a union of the two queries.  If you open the Advanced section for your report, you can see the generated query. For more information on reversible relationships and the metadata associated with them, see the Jazz.net article Integrating external data sources with LQE and Report Builder.

Scoping configurations at run time

When running a report against the “LQE scoped by a configuration” data source, you must specify a configuration context in the Choose a Configuration runtime filter.

To provide the list of configurations, RB queries LQE for configurations that are directly or indirectly associated with the project areas and components selected by the report, and based on user access permissions. The filter displays the configurations in a 4-level hierarchy:

  1. Projects, including projects from the GCM application
    • Project areas that contribute at least one configuration or have at least one associated global configuration.
    • If a configuration does not provide enough information to determine the associated project, RB lists it under a Streams entry at the same level as the projects.
    • If the user selects projects in the Limit Scope filter, the list of components and configurations is scoped based on those selections.
  2. Components in the project area (versions earlier than 6.0.3 do not support components and omit this level of the hierarchy)
    • Each project entry expands to show the list of components defined for that project.
    • If the user selects components in the Choose a Component filter, the list of available configurations is scoped based on those selections.
  3. Streams in the component
    • Each component expands to show the list of stream configurations for that component, including global configurations.
  4. Baselines of the stream
    • Each stream expands to show all the baselines created on that stream.

Selecting a component in the Choose a Component runtime filter does not scope the results for the report, only the list of configurations shown in the Choose a Configuration filter. To scope the report results, set a condition in the report using the Component attribute for one or more of the artifacts in the report.

For more information on using Report Builder to build and run reports, refer to the IBM Knowledge Center and the tutorials included with Report Builder.

Additional resources

The following resources provide more information on LQE and Report Builder:

Appendix: Understanding merged resource shapes

This appendix describe how LQE and RB merge resource shapes across project areas to better support cross-project reporting. These topics are of interest primarily to developers enabling applications to contribute data to LQE, and to users building advanced LQE queries. Most users and administrators can skip these sections.

Resource shapes, defined in the OSLC Resource Shape (2.0) specification, provide a way to define all the possible attributes for a data type. In general, the concept URI for a resource shape is specific to each project. Since most resource shapes use anonymous oslc:Property resources, their metadata representation is also unique to the project. The user might see two projects that use an equivalent resource type, but from the metadata perspective, those types and their properties are completely different. When Report Builder builds its metamodel, it would interpret and expose those types as two separate and distinct types. For example, if two RM projects each define a Special Requirement artifact type, report creators would see two different artifact types in RB, each with its own set of attributes. This complicates reporting across projects.

To address this issue, when applications publish appropriate metadata, LQE runs merging queries to construct identical URIs for equivalent resource shapes, and for equivalent properties of a given resource shape. When an application publishes a resource shape graph to the main LQE index, LQE publishes an equivalent “merged” resource shape graph with deterministic URIs to the “shapes” partition, as described below in How LQE publishes merged shapes. Resource shapes with the same name and domain would have the same deterministic URI, indicating equivalence.

Because the equivalent shapes and properties have identical URIs, RB then combines all the triples for each resource type or property, and creates a merged version of that type or property in the metamodel, representing a superset of the properties and property values for that type. For example, if one RM project defines a RequirementA type with custom properties abc and xyz, and another defines a RequirementA type with custom property def, and publishes the resource shapes for those types correctly, the RB metamodel would show a merged RequirementA type with any common properties as well as all the custom properties (abc, xyz, and def).  When RB merges the triples for a property, it detects any conflicts and indicates those in the user interface.

Shape merging also takes access control into account. The TRS feeds for resource shapes and properties include metadata indicating the project area that defines those elements. LQE copies that metadata (oslc:serviceProvider) into the merged shape and all of its properties. When RB builds its meta-model, every type and property includes a list of all the projects that published them. RB filters the types and properties based on the user’s project permissions (as described in the section above) and any selections made in the Limit Scope section.

The sections that follow describe resource shape merging in more detail. The Jazz.net article Integrating external data sources with LQE and Report Builder describes the metadata required in application TRS feeds to support the merge operations.

How LQE merges shapes

When a TRS feed publishes a resource shape to LQE, LQE copies it into a special “shapes” partition, generating a deterministic URI with the format:   

https://jazz.net/ns/lqe/merge/gensym/domain/shapeName

where shapeName is the value of the resource shape’s dcterms:title property, and domain is the domain associated with its TRS feed, either by the feed’s dcterms:type property (am, auto, cm, qm, rm, or config), or specified by an administrator when manually adding a TRS feed.  For example, if you had two RM project areas that each define a resource shape type RequirementA, the merged resource shape URIs would be identical; similarly, if you have multiple instances of the RM application, they would define the same TRS providers, and equivalent types from both instances would have identical merged shape URIs.

LQE also merges the properties for the resource shape based on the URI for the merged shape and the property predicate URI (oslc:propertyDefinition value) of the property:

https://jazz.net/ns/lqe/merge/gensym/domain/shapeName/propertyPredicateURI

Because the oslc:propertyDefinition value is already a URI, LQE encodes the value before adding it as the propertyPredicateURI. 

If the application publishes versioned resource shapes, LQE creates non-versioned graphs in the “shapes” partition using the merged shape URI format described above. When RB queries the “shapes” partition, the triples defined for the non-versioned and versioned resource shapes are automatically combined in the query results.

Note for 6.0.1 users: In version 6.0.1, LQE used the URI of the TRS feed to determine the domain string; for example a resource from the TRS feed https://myhost.com:9443/ccm/oslc/workitems/trs would have a merged shape URI like  https://jazz.net/ns/lqe/merge/gensym/ccm%2Foslc%2Fworkitems%2Ftrs/shapeName. If you are using 6.0.1 and have multiple instances of the same application, shape merging will only work if all instances use the same context root in their TRS feeds.

How LQE merges shape properties

In the merged shape, all of the oslc:xxx predicates used to define properties for a resource shape are converted to merge:xxx predicates, with the exception of standard properties like dcterms:title and acc:accessContext, which LQE uses for other purposes. This is done to avoid confusion between a resource shape in the main partition and its merged shape in the shapes partition.

Similarly, LQE converts individual oslc:allowedValue properties to merge:allowedValue properties. If the resource shape property instead uses oslc:allowedValues to refer to a resource of type oslc:AllowedValues, the merging process flattens that referenced resource and inserts the individual merge:allowedValue statements directly into the property definition itself.

In the merged shape triples, the URIs in the object part remain unchanged. If the object URI refers to a resource that doesn’t exist in the shapes partition, RB cannot locate the referenced resource when it reads the metadata. To address this issue, when it loads the metadata, RB converts certain object URIs, such as the value for oslc:valueShape, from the original resource shape URI to the corresponding merged shape URI.

How RB finds resources corresponding to a merged shape

The metadata published for an instance resources (an actual artifacts you would report on) includes an oslc:instanceShape property that points to the appropriate project-specific resource shape; it does not refer to any merged shape types. RB only reads and exposes the metadata for merged shapes; when the user selects a merged shape type, RB knows the correct merged shape URI but not the individual resource shapes that contributed to that merged shape.

To connect the individual resource shape to the merged shape, LQE injects a merge:mergeShape property into each published resource shape, referring to its merged shape. RB uses this property and the merged shape URIs to find the resources corresponding to the known merged shape, generating SPARQL like: 

?resourceUri oslc:instanceShape ?resShapeUri.
?resShapeUri merge:mergeShape <knownMergedShapeURI>.

By default, RB loads only the merged shapes from the “shapes” partition. If for some reason RB can’t find the endpoint for the “shapes” partition, it will load all the individual project-specific resource shapes;this could impact performance as well as usability for the report author (who must choose from a much longer list of artifact types).

For more information on resource shapes

Also see the Additional resources listed earlier in this article.

About the authors

Kevin Cornell is a Senior Software developer who played a significant role in the development of Report Builder and JRS. Kevin recently retired. 
Kathryn Fryer is a Solution Architect who works closely with clients, field, and development to develop usage models, aid adoption, and improve product offerings. Kathryn can be contacted at fryerk@ca.ibm.com.

© Copyright IBM Corporation 2017

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Mon, 09 Apr 2018