Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 2: Automation
This article assumes that you are familiar with the these concepts:
- Collaborative Lifecycle Management (CLM) and Continuous Engineering (CE) tool support for configuration management
- How to work with Rational DOORS Next Generation configurations and perform common operations such as selecting a configuration and delivering changes
If you are unfamiliar with these concepts, see the Configuration management overview and Managing artifacts by using configurations sections in IBM Knowledge Center. Also, ensure that you are familiar with the information in Part 1 of this series.
Contents
- Automated procedures for maintaining Rational DOORS Next Generation type systems for editable configurations
- Type system management prototype
- Type system management prototype: Manual mapping
- Type system management prototype: Automated mapping
- Summary
Automated procedures for maintaining Rational DOORS Next Generation type systems for editable configurations
Part 1 of this series emphasizes the importance and benefits of maintaining consistent Rational DOORS Next Generation type systems. However, maintaining consistency can be challenging for large Rational DOORS Next Generation implementations. Organizations might have hundreds of components and thousands of editable Rational DOORS Next Generation configurations. A change to an organization’s common Rational DOORS Next Generation type system definition must be applied to each component, and then to each editable configuration. For with any substantial number of components and configurations, success depends on well-defined maintenance approaches that are performed consistently.
Regardless of how well defined the processes are, the manual management of hundreds of components and thousands of streams and change sets is difficult, error prone, and expensive. The question is how to ensure that changes are delivered to all required targets. Has a change been omitted? Was a wrong target selected? How can the process be documented and tracked? How can the process be reproduced after the next change?
Cost is also an important concern. Suppose there are 10 components and an average of 50 streams per component, and that delivering a type system change to a stream takes a user only a minute. Manually delivering the type system changes for all components to all streams would take at least 8 hours.
For all these reasons, you should consider automating the change process. With automation, you can repeat, document, and track the process. Automation also scales better.
The following images show two scenarios that require automation. The following image shows a logical view of a type system import operation, from one stream in a component into another stream in another component. 
The following image shows a logical view of a type system deliver operation, where a change on one stream is delivered to another stream in the same component. 
Type system management prototype
A Rational DOORS Next Generation type system management prototype was developed to automate these operations. The goal of the prototype was to understand the required APIs, how to use them, and to better understand the problem domain and the use cases. This article describes the resulting prototype, how it works, which APIs are required, and how they were used. Version 1.0 of this type system management prototype provides the following commands:
- exportConfigurations: For all components owned by a project area, export all stream configurations into a comma-separated values (CSV) file.
- exportConfigurationsByDescription: For all components owned by a project area, export a source-to-target mapping table based on corresponding tag sub-strings in the stream description into a CSV file.
- exportAllConfigurationsByDescription: For all components owned by any project area, export a source-to-target mapping table based on corresponding tag sub-strings in the stream description into a CSV file.
- importTypeSystem: Read a CSV file with a mapping of source component stream configurations to target component stream configurations to import the type system of the source configuration into the target configurations.
- deliverTypeSystem: Read a CSV file with a mapping of source component stream configurations to target component stream configurations to deliver the type system of the source configuration to the target configuration.
The format of the CSV output from the exportConfigurations and exportConfigurationsByDescription commands can be used for the importTypeSystem and for the deliverTypeSystem commands. Each row in the CSV file used as input for the importTypeSystem or the deliverTypeSystem operation describes one source-to-target mapping for the operation.
The exportConfigurations and the exportConfigurationsByDescription commands currently run against one project area to find the configurations. You can export configurations for all project areas.
The exportAllConfigurationsByDescription command exports configurations for all configuration-management-enabled project areas.
The importTypeSystem and the deliverTypeSystem commands assume that the source configuration, typically a stream, has changes in the type system. To perform such changes, a change set is created. The required changes are performed on the type system and the change set is delivered to the stream.
The importTypeSystem command assumes that a change set is required for any changes. The command creates a change set for the target stream. The change set is used to import the type system changes. Then, the change set is delivered to the target stream.
The deliverTypeSystem command assumes that changes from the source configuration can be delivered to the target configuration.
The importTypeSystem and the deliverTypeSystem commands fail if no changes exists between the source and target configuration. The API provides limited information about the failure, so it is difficult to tell why the error occurred. An enhancement request has been submitted to provide better feedback in the API to distinguish the failure cases.
The importTypeSystem and the deliverTypeSystem commands are not limited to configurations of one project area. As long as the combination of source and target configuration is supported, the mapping can be used and the operation executed. To successfully perform the operation, the user ID provided to the command must have access to all the objects and the permissions required for the operation.
One result of the prototype was the reinforcement that manual maintenance of the CSV files is tedious and error prone. The exportConfigurationsByDescription command removes manual maintenance of the mapping file. The limitation to only one project area is impractical for bigger deployments, so the the exportAllConfigurationsByDescription command removes the limitation.
Commands and syntax
Running the type system management tool without a command prints the available commands and the commands syntax. The commands and their syntax are as follows:
exportConfigurations
-command exportConfigurations -url <public URI> -user <user Id> -password <password> -project <Project Area Name> -csvfile <destination FilePath> [ -csvDelimiter <delimiter> ]
Example:
-command exportConfigurations -url https://clm.example.com:9443/rm -user ralph -password ralph -project “GC JKE Banking (Requirements Management)” -csvfile exporttest.csv -csvDelimiter “;”
exportConfigurationsByDescription
-command exportConfigurationsByDescription -url <public URI> -user <user Id> -password <password> -project <Project Area Name> -csvfile <destination FilePath> [ -csvDelimiter <delimiter> ]
Example:
-command exportConfigurationsByDescription -url https://clm.example.com:9443/rm -user ralph -password ralph -project “GC JKE Banking (Requirements Management)” -sourceTag TSSource_TS1 -targetTag TSTarget_TS1 -csvfile exporttag.csv -csvDelimiter “;”
exportAllConfigurationsByDescription
-command exportAllConfigurationsByDescription -url <public URI> -user <user Id> -password <password> -csvfile <destination FilePath> [ -csvDelimiter <delimiter> ]
Example:
-command exportAllConfigurationsByDescription -url https://clm.example.com:9443/rm -user ralph -password ralph -sourceTag TSSource_TS1 -targetTag TSTarget_TS1 -csvfile exporttag.csv -csvDelimiter “;”
importTypeSystem
-command importTypeSystem -url <public URI> -user <user Id> -password <password> -csvfile <destination FilePath> [ -csvDelimiter <delimiter> ]
Example:
-command importTypeSystem -url https://clm.example.com:9443/rm -user ralph -password ralph -csvfile TypeSystemImport.csv -csvDelimiter “;”
deliverTypeSystem
-command deliverTypeSystem -url <public URI> -user <user Id> -password <password> -csvfile <destination FilePath> [ -csvDelimiter <delimiter> ]
Example:
-command deliverTypeSystem -url https://clm.example.com:9443/rm -user ralph -password ralph -csvfile TypeSystemDeliver.csv -csvDelimiter “;”
All commands support the optional parameter -csvDelimiter, which enables you to control the delimiter used in the CSV file. The default delimiter is a comma (,). Use -csvDelimiter “;” if you intend to use Microsoft Excel to edit the CSV file. Excel uses a semicolon (;) as default delimiter in the CSV file format. Using a semicolon as a delimiter enables you to double-click the file in Windows Explorer to correctly open it in Excel. Use a convenient delimiter for the application of your choice.
Configuration management
The type system management prototype was developed to support the best practices described in Part 1. Limitations of the available API and limitations due to the scope of the prototype implementation lead to the following assumptions and limitations.
The deliverTypeSystem command assumes that the source stream contains changes compared to the target stream. The source stream is delivered against the target stream. Currently no change set is created and the delivery works even if a change set is required on the target stream. The delivery works even if no change set was created in the source stream. This behavior might change in later versions.
The importTypeSystem command assumes that the source stream contains changes compared to the target stream. To ensure the operation always works and to follow best practices, a change set is created for the target stream. The source stream is delivered against the change set and the change set is then delivered to the target stream.
Note: The deliverTypeSystem and the importTypeSystem commands fail if they are rerun without a change in the source type system. The currently available API does not allow to detect if changes are available and the API provides only limited information about the failure, so it is difficult to tell why the error happened.
The type system maintenance commands in the prototype are currently unable to create or link an approved change request or work item to a change set. This means that the To deliver to this stream, change sets must be linked to an approved change request or work item option is not currently supported for the importTypeSystem command. If this option is selected on a target stream, the type system import command creates a change set, but the import fails because of the missing approval work item.
With respect to this option, the only scenario that fails is the type system import to a stream that requires a change set with a work item. No other scenario of a change set and approval requirement on a source and target stream should fail.
The importTypeSystem command creates change sets to deliver the changes. If the deliver operation fails, the change sets created by the application are discarded.
APIs used for the prototype
To implement the prototype functionality, the following APIs were used.
Basic public OSLC-based REST APIs are used to discover components and streams. Special public APIs are used to perform the subsequent necessary configuration-management-aware operations. For the main entry point for these server APIs, see the IBM Rational DOORS Next Generation Server API documentation.
After the source and target configurations are located, the following APIs are used:
- DNG Type Import: Import types from one component or project to another
- DNG Configuration Management: Enables working with DNG Delivery Sessions
- DNG Task Tracker: Task trackers enable monitoring of asynchronous long-running server activities
- An API to create change sets is supported but no public documentation is available
- An API to discard change sets is supported but no public documentation is available
Configuring the runtime environment
The examples in this article are based on this configuration:
- An up-to-date and compatible Java Runtime Environment (JRE) version 1.8 is installed and in the system path
- The type system management prototype is packaged as a file named tsm.jar in the c:\temp\TSM folder
- For logging output, a valid log4j.properties file is in the same file path location
For details, see Part 3 of this series.
Type system management prototype: Manual mapping
Initially, the prototype followed this usage pattern:
- Export a source-to-target mapping file
- Modify the mapping file as needed
- Run an operation by using the mapping file
- Maintain the source-to-target mapping file
1. Export a source-to-target mapping file
Start by exporting the streams and configurations of a project area. Currently, the prototype is limited to a project area for the export. You can change the scope in the future, but more data makes it difficult to manage and find what you are looking for.
To collect information about the configurations, run the exportConfigurations command. Example:
java -jar tsm.jar -command exportConfigurations -url https://clm.example.com:9443/rm -user ralph -password ralph -project "GC JKE Banking (Requirements Management)" -csvfile "../exporttest.csv" -csvDelimiter ";"
The first time this command runs, it creates the initial source-to-target mapping file that is used as input for the automation. In later runs, the source-to-target mapping file is reused and modified.
2. Modify the mapping file as needed
Open the c:\temp\exporttest.csv file with an editor. The following image shows a Microsoft Excel file. Examine the file.
The following columns are created during the export operation:
- source: The URI of the configuration used as the source in the source-to-target mapping file used in the importTypeSystem or deliverTypeSystem commands.
- target: The URI of a configuration (currently only streams) that was found in the project area. This URI uniquely identifies the stream used in the automation. A typical URI has this format: https://clm.example.com:9443/rm/cm/stream/_D8BRZtNwEei8yOyAL-l3PQ.
- targetTitle: The title of the stream, not necessarily unique; provided only for usability and is not used in the automation.
- targetDescription: The description of the stream, not necessarily unique; provided only for usability and not used in the automation.
- targetComponentName: The name of the component the target stream belongs to, not necessarily unique; provided only for usability and not used in the automation.
- projectAreaName: The unique name of the project area, provided only for usability and not used in the automation.
The automation needs only the source and target configuration URIs. The user who creates the source-to-target mapping file needs the information in the other columns.
After the data is exported, you must define the source configuration or stream for the type system, with respect to the planned importTypeSystem or deliverTypeSystem operation. This example uses the deliverTypeSystem operation, but the process is the same for the importTypeSystem operation.
To define the source configuration or stream for the type system, you must identify the stream used to develop the type system. Any changes to the type system are performed here and delivered to this stream. This stream always contains the latest version of the type system. Changes to this stream must be delivered to other streams that use the same type system.
In this example, the stream is named Type System Delivery Development Stream and is shown in row 8. The name describes its purpose. The target description shows its purpose, too: Type System TS1 Source Stream.
The next step is to identify the streams that the type system must be delivered to, if there is a change. These are the streams in rows 9-11. The names contain a hint, for example, Type System Delivery Work Stream 1 and the description contains another hint: Type System TS1 Target Stream. In a real environment, you must know which streams to select.
To make this stream the source configuration, copy the URI of the stream into the source column in front of the streams that are the target for the deliver operation. In the current example, copy the URI in the target column from cell B8 into the source column for rows 9-11 into the cells A9, A10, A11. The result should look similar to the following image:
Finally, delete all lines without a source column. These columns are not needed, because the configurations are not relevant to the operation. If there is another source stream and another set of target streams for managing a different type system, you would manage them in a different CSV file. Although you could manage them by using another set of rows, the clutter and current API makes this impractical.
Save the source-to-target mapping file as C:\temp\TypeSystemDelivery.csv. This file will be used for subsequent operations. The file is updated when new streams are created that require maintenance.
3. Run an operation by using the mapping file
You can now use the CSV file to run the type system change operation such as importTypeSystem or deliverTypeSystem. This example shows the deliverTypeSystem operation. In the command window, type the following command and press Enter:
java -jar tsm.jar -command deliverTypeSystem -url https://clm.example.com:9443/rm -user ralph -password ralph -csvfile "../TypeSystemDelivery.csv" -csvDelimiter ";"
The output of the operation looks similar to the following image: 
Note: The deliverTypeSystem and the importTypeSystem commands will fail if you run them again without a change in the source type system. The API provides only limited information about the failure, so it is difficult to tell why the error occurred.
4. Maintain the source-to-target mapping file
Over time, new streams using the same type system are created. You must add these streams to the CSV file that performs the type system change operation. To do this, run the exportConfigurations operation again, as shown in this example:
java -jar tsm.jar -command exportConfigurations -url https://clm.example.com:9443/rm -user ralph -password ralph -project "GC JKE Banking (Requirements Management)" -csvfile "../exporttest.csv" -csvDelimiter ";"
Identify new rows in the exported configuration data in c:\temp\exporttest.csv. Copy the rows for new configurations that need to be included into the related resource-to-target mapping file such as C:\temp\TypeSystemDelivery.csv. Copy the source configuration into the source column in front of the new lines. The new configurations are now included in the type system change operation and will be in subsequent runs.
The tool ignores archived streams; however, you should remove archived streams from the mapping files.
Type system management prototype: Automated mapping
Based on early iterations of the prototype, the manual maintenance of the mapping file was tedious and error prone. A more automated creation of the source-to-target mapping would be better. So, more automation was added, which led to the following usage pattern:
- Manage mapping information in the stream description
- Automatically create the source-to-target mapping file
- Run an operation using the mapping file
- Repeat the operation or re-create the mapping file if needed
1. Manage mapping information in the stream description
The source-to-target mapping could be enhanced by using meta-information. Currently, Rational DOORS Next Generation does not provide any way to store information such as tags or custom properties for streams. However, until meta-information is available, you can use the description to store this type of information. Assume the following conditions are true:
- On the server, the type systems TS1..TSn are maintained.
- The type systems are maintained in a stream, one for each.
- The type systems are used in many streams.
Assume the following business rules:
- In each TS1 .. TSn type system maintenance stream, ensure that the following tag string is part of the description: TSSource_TS1 .. TSSource_TSn
- In each stream that uses a TS1 .. TSn type system, ensure that the following tag string is part of the description: TSTarget_TS1 .. TSTarget_TSn
If you consistently apply and maintain the tags, you can search the description for tags and generate the mapping table automatically. You can also search for streams that do not follow the convention.
The following image shows a simple scenario.
2. Automatically create the source-to-target mapping file
Based on this idea, an additional exportConfigurationsByDescription command was developed that uses this information to generate a source-to-target mapping. The following command generates the same CSV source-to-target mapping file, but does not require any manual work:
java -jar tsm.jar -command exportConfigurationsByDescription -url https://clm.example.com:9443/rm -user ralph -password ralph -project "GC JKE Banking (Requirements Management)" -sourceTag TSSource_TS1 -targetTag TSTarget_TS1 -csvfile "../exporttag.csv" -csvDelimiter ";"
However, the command relies on tag information in the stream description.
The initial limitation to export configurations from only one project area was removed by providing the exportAllConfigurationsByDescription command. This command exports a source-to-target mapping file for all streams, for all components, for all configuration-management-enabled projects. The following command generates the same CSV source-to-target mapping file across all project areas:
java -jar tsm.jar -command exportConfigurationsByDescription -url https://clm.example.com:9443/rm -user ralph -password ralph -sourceTag TSSource_TS1 -targetTag TSTarget_TS1 -csvfile "../exporttag.csv" -csvDelimiter ";"
3. Run the type system change operation
You can now use the CSV file to run the importTypeSystem or deliverTypeSystem type system change operation. This example shows the deliverTypeSystem operation. In the command window, type the following command and press Enter:
java -jar tsm.jar -command deliverTypeSystem -url https://clm.example.com:9443/rm -user ralph -password ralph -csvfile "../exporttag.csv" -csvDelimiter ";"
4. Repeat the operation or re-create the mapping file if needed
Repeat the type system change operation when new changes are made to the type system. To adjust for newly created streams, re-create the mapping file.
Summary
This article shows that automating the management of Rational DOORS Next Generation type systems is possible. The first experiences with the tool showed possible enhancements which where already implemented. Using the base code, you can further automate these tasks. A possible enhancement would be to avoid using the CSV file and run operations based directly on tags in stream descriptions.
Plans are in place to publish open-source code and articles that explain how to use it.
Comments, feedback, ideas, and experiences are appreciated. If you have questions, ask them in the Jazz.net forum instead of commenting on this article. Tag the question as a Rational DOORS Next Generation question and add this tag: dng-type-system-management
Related information
- Maintaining the Rational DOORS Next Generation type system in a configuration-management-enabled environment. Part 1: Manual procedures
- Rational DOORS Next Generation server APIs
About the authors
Ralph Schoon is a member of the IBM Watson IoT Unleash the Labs team, which has a mission to help IBM staff and customers apply Jazz technologies. Ralph has his own blog about extending and automating Rational Team Concert. He has published multiple articles on Jazz.net and is committed to working with Jazz users on the Jazz.net forums to help them understand the technical details of the Jazz products and optimize their solutions. Before his current role, Ralph worked as a Leading Technical Sales Representative for IBM, a team lead and developer for machine control for an industrial company, a developer, a Microsoft Certified Professional, and a system and network administrator. He can be contacted at ralph.schoon@de.ibm.com.
Todd Dunnavant, Ph.D., P.E., is the Continuous Engineering Practice Lead and a Principal Solution Architect with IBM Watson IoT Services. He has 20 years of experience successfully transforming how software, systems, and product engineering teams work. For the past 7 years, Todd has focused on realizing these transformations using the IBM Jazz-based products. More recently, Todd has helped multiple product-centric companies adopt the lifecycle configuration management capabilities of the IBM Internet of Things Continuous Engineering tool suite. He can be contacted at twdunnav@us.ibm.com.
Ian Zimmermann is a CLM Solution Architect who has led requirements management, CLM, and global configuration implementations for many years. A former aerospace engineer with the Canadian Military, Ian has also held roles including Director of IZTech (an IBM Business Partner) and Director of Requirements and Project Portfolio Management for Telelogic. With deep expertise in the CLM suite and Rational DOORS, Ian is a Certified Rational DOORS Next Generation expert and has successfully migrated over 75 databases from Rational DOORS to Rational DOORS Next Generation. He can be contacted at ianz@ca.ibm.com.
© Copyright IBM Corporation 2019