EditAttachPrintable
r2 - 2016-04-28 - 23:44:55 - TimFeeneyYou are here: TWiki >  Deployment Web > CLMExpensiveOperations > ScenariosToBeInvestigated
e

CLM Operations to Investigate as Potentially Expensive(DRAFT) uc.png

Authors: TimFeeney
Build basis: The Rational solution for Collaborative Lifecycle Management (CLM) and the Rational solution for systems and software engineering (SSE) v6.0.1.

Page contents

This page captures CLM operations that may potentially be expensive but need further investigation to characterize and confirm. See the parent topic containing the known list.

Note: The information herein is a work in progress and is continually being reviewed/refined

Table 1: Summary of CLM Operations to Investigate
Product Scenario Scenario ID Best Practice Investigate
Rational DOORS Next Generation (DRAFT) Enabling of Suspect links NA link NA
Running large RPE/RRDG reports DNG_Report link NA
Performing large imports DNG_Import link NA
Performing large exports DNG_Export link NA
Using large view queries DNG_Query link NA
DNG ETLs NA link NA
 
Rational Team Concert (DRAFT) Concurrently loading a large number of large configurations RTC_Load_Workspace link TBD
Comparing extremely out of date repository workspace with stream RTC_Compare_Workspace link 389280
Annotating an extremely large text file RTC_Annotate_File link NA
Running a query that returns a large number of artifacts RTC_Query link NA
Import of large Microsoft Project plans RTC_MSP_Import link 389355
Import of large CSV files RTC_CSV_Import link 389357
Export of large Microsoft Project plans RTC_MSP_Export link 389358
Export of large number of work items to CSV file RTC_Workitem_Export link 389359
Large bulk edits of work items RTC_WI_Bulk_Edit link 389361
Adding many build result contributions to build result RTC_Add_Build_Contribution link 389278
Loading a large plan RTC_Load_Plan link 389362
 
Rational Quality Manager (DRAFT) Duplicating test plans with a large test hierarchy. RQM_Duplicate_Test_Plan link NA
Bulk archiving and/or deletion of test results at end of release. RQM_Bulk_ArchiveDelete link NA
 
Jazz Reporting Services (all reporting technologies) (DRAFT) BIRT reports based on live data JRS_BIRT link NA
Reports with more results or attributes than necessary NA link NA
Large reports on frequently-accessed dashboards NA link NA
Data aggregation and trend reports with large time intervals NA link NA
DCC jobs requiring high storage and processing power NA link NA
LQE index maintenance NA link NA
High volume and highly complex queries JRS_LQE_Query link NA
Refreshing a data source in Report Builder JRS_Refresh_Data_Source link NA
 
Global Configuration (DRAFT) Creating streams in a global configuration hierarchy GC_Create_Stream link 389494
Creating a baseline staging stream from a global stream hierarchy GC_Create_Baseline_Staging link 389497
Creating a global baseline (or committing a baseline staging stream) from a global stream hierarchy GC_Create_Global_Baseline link 389495

Rational DOORS Next Generation

Enabling of Suspect links.

Link validity and suspect links are capabilities used to monitor related information for updates. When dependencies change (such as a customer requirement) we need a way of marking related information as "suspect" so that it can be reviewed and updated as appropriate.

Suspect links and link validity offer very similar functionality but under very different circumstances. Suspect links describe related information within a single context whereas link validity is used when Configuration Management functions are being used.

Due to the approach used for local indexing for suspect links you should be aware that there is resource overhead on the server that you should consider when deploying that functionality. Its performance is a function of the number of artifacts, concurrent users and the rate of change activity. Often it is turned on without understanding what it does, how it will be used or the cost of doing so.

Best Practices for Enabling of Suspect links.

Regular use in production during normal hours should be limited to small deployments, when the server is lightly loaded, or only when necessary, otherwise you risk driving load on the server.

Once suspect tracking is enabled, a full reindex occurs up front (see Suspicion Indexing). When you enable suspect traceability (in a project that is not enabled for configuration management), an index of change information for all link types, artifact types, and attributes is automatically built. It should not be turned off then on again as that will cause another full reindex. Instead, pause the indexing (via the Suspicion Profile Settings for Requirements Management page).

By default, the index is automatically refreshed with new changes every 60 seconds, but you can change that setting. If you lower the default refresh setting, a greater load is placed on the server. For the refresh setting, use a number that is 30 seconds or higher.

This does not apply to projects with configuration management enabled, in which case link validity, not suspicion profiles, provides the capability to note changes in requirements.

Running large RPE/RRDG reports.

Generation of poorly constructed RPE/RRDG reports that include a result set of over 5K requirements can be slow. Concurrent generation of PDF reports for modules including more than 5K requirements should be limited to generating only one PDF at a time. If the report size can't be pre-estimated, then all PDF report generation should be limited t o TBD reports at a time.

Best Practices for Running large RPE/RRDG reports.

Be sure to test reports in advance with near production equivalent data. Where possible, define the report with JRS.

Performing large imports.

Importing can be computationally expensive. At the end of an import, when all indexing occurs, the import can block other user activity. Imports of 10K requirements or less should be fine. DNG currently limits how many ReqIF imports occur at once through the ReqIF.importThreadPoolSize advanced property.

Best Practices for Performing large imports.

For large (10K or greater) imports, recommend importing during off hours or when the system is lightly loaded.

Performing large exports.

Export is less of a problem with CSV/ReqIF exports. Note that the number of concurrent ReqIF exports is limited to the number of virtual cores on the DNG server. Word/PDF can be problem and should be limited to one at a time when exporting >10K requirements.

Best Practices for Performing large exports.

Recommend exporting during off hours for large (10K or greater) exports to Word/PDF.

Using large view queries.

Large queries resulting in 10K or more requirements can be expensive, especially across multiple folders or filtered based on strings, dates or links. Traceability queries are the more expensive patterns. DNG has the Advanced Property 'query.client.timeout' to limit the run time of view queries; it defaults to 30 seconds. There is a second property "SPARQL Query abort timeout (in ms)" that will limit most queries not limited by 'query.client.timeout' (e.g., loading folder structure).

Best Practices for Using large view queries.

See DNG Queries Best Practices).

DNG ETL's.

This is a more of a system initiated operation and is a function of repository size and amount of change. Generally for repository sizes >100K-200K it is best to run the ETL jobs off hours. Alternatively, use DCC to populate the data warehouse. This doesn't apply to configuration management enabled projects that populate the LQE database instead of the data warehouse.

Best Practices for DNG ETL's.

For large repository sizes, containing >100K-200K artifacts, it is best to run the ETL jobs off hours.

Rational Rhapsody Design Manager

Importing or moving a large model to DM

When you import a large model, or move a large model (in actively-managed mode) to DM, the server could potentially become unresponsive, especially when others are working in that same stream. DM notifies users that an import is occurring.

Operations that load a large model

Some operations require loading a model in its entirety; if the model is large, it can impact server memory and processing. Such operations include:
  • Generating code from a model
  • Exporting a model (using “Save as”)
  • Creating a table layout in Rhapsody with too broad a scope

Use smaller related models to reduce the system demands; this is also a best practice.

OSLC integration to DNG/DOORS

Rhapsody DM can manage OSLC links between model elements and requirements artifacts in related DOORS and DNG projects. When doing so, DM retrieves all available requirements from the projects and all defined links from the DM server. The integration synchronizes to find new requirements/links. If the requirements project is large, the link management can be resource-intensive.

To reduce resource demands, use a filtered view (e.g. collection, module) to narrow the amount of requirements to retrieve.

Rational Team Concert

Concurrently loading a large number of large configurations

This may occur with many continuous builds doing many concurrent loads of large configurations (those approaching the file/folder limit as described in RTC Limits).

This scenario needs further analysis to confirm it can be expensive and to characterize further. See work item 389280.

Comparing repository workspace with streams when the workspace is extremely out-of-date with the stream

This could cause server issues if a large number of these types of comparisons happen concurrently. As of 5.0.1, a server property may be set that limits the number of comparisons that can happen at the same time (Maximum limit of SCM workspace compare operations).

Annotating an extremely large text file can cause out of memory

This is a point in time limitation in 6.0.2 that is expected to be addressed in a future release. See OutOfMemoryError when trying to annotate a 2G file. The expected default of 1MB may be adjusted as further testing is performed. The issue has been observed when annotating files with size in the multi-gigabyte range. Since the annotate operation puts all contents, file, history, etc, in memory, large files, especially with large history, can require significant memory.

Running a query that returns a large number of artifacts

By default query results are capped at 1000 work items (by the advanced server property Maximum Query Result Set Size). At this number, performance should be fine. Increasing the setting above 1000 could impact server performance, especially if the query is included on a frequently-accessed dashboard tab. If you encounter issues, reduce the Maximum Query Result Set Size. If you use Oracle as your RTC database, there is a known issue with these large queries. See No items to be selected if using Oracle and Query Result Set Size is set to 1001..

Best Practices for Running a query that returns a large number of artifacts

Should it be absolutely necessary to run a query that would return greater than the default maximum, run them instead using OSLC queries during off hours or when the system is more lightly loaded. See Consuming Rational Team Concert’s OSLC Change Management V2 Services.

Import of large Microsoft Project plans

How long an import takes depends on the quantity of items in the plan and their nested structure. For example, an import from an MS Project file containing 2000 tasks could take up to 30 minutes on the first import and 8-10 minutes on subsequent imports depending on server configuration and load. Consider also the memory demands of an import which will take approximately 100KB for each task being imported over and above the memory needed for typical RTC operation. In most cases, import of Microsoft Project plans is an infrequent operations, generally performed at the start of a project. However, if imports are to be a frequent occurrence, be sure that the server memory allocation has ample spare capacity. Note the numbers provided are based on testing in a non-production environment.

Further investigation to better characterize a 'large' Microsoft Project plan and the load generated when importing one is tracked by work item 389355.

Best Practices for Import of large Microsoft Project plans

If your MS Project file contains more than 1000 tasks, we recommend you import or export during off-hours or when the server is lightly loaded.

Import of large CSV files

Similar to an import of a Microsoft Project plan, an import of a large CSV file can keep the server busy and drive memory usage. However, unlike the plan import, CSV file imports occur more frequently and are often part of a round trip update.

Further investigation to better characterize a 'large' CSV file import and the load generated when importing one is tracked by work item 389357.

Best Practices for Import of large CSV files

For more than 1000 work items, it is best to wait for off hours or until the server is lightly loaded.

Export of large Microsoft Project plans

Similar to an import, export time and load are dependent on the size and complexity of the plan. The impact is primarily to memory on the server.

Further investigation to better characterize a 'large' Microsoft Project plan and the load generated when exporting one is tracked by work item 389358.

Export of large number of work items to CSV file

Exporting a large number of work items primarily impacts memory on the server.

Further investigation to better characterize a 'large' number of work items and the load generated when exporting them is tracked by work item 389359.

Large bulk edits of work items

Performance and load driven by this scenario is a function of the number of work items and number of attributes changed as each work item is changed and saved one at a time.

This scenario needs further analysis to confirm it can be expensive and to characterize further. See work item 389361.

Adding many build result contributions to build result

When a large number of contributions (compilation contributions, Junit tests, downloads, log files, links, work item references, etc) are included on a build result, because of the way they are stored (single data structure), the server could spend a lot of time marshalling and unmarshalling the persisted contributions when adding/deleting contributions. At best this is a slow running operation, however, should there be a large number of concurrent builds performing similar work (adding many build result contributions), the potential for impact to the server increases.

This scenario needs further analysis to confirm it can be expensive and to characterize further. See work item 389278.

Loading a large plan

The RTC plan editor provides users with great flexibility to display plans in customized and flexible configurations. In order, to provide rapid display of custom plan configurations, the RTC planning editor must fetch all the details of each work item when loading plans. Consequently, when the scope of a plan includes a large number of work items, loading of such plans can drive server load. We have greatly improved plan loading performance with each release by deferring the loading of out placed "child" work items or by allowing users to turn on and configure server side plan filtering to avoid loading plan types that will never be displayed in plans.

This scenario needs further analysis to confirm it can be expensive and to characterize further. See work item 389362.

Best Practices for Loading a large plan

For large plans using the traditional RTC plan UI, consider configuring the plan to defer loading of child work items and perform filtering on the server.

In addition, later releases of RTC (5.0.2 and forward) include a new planning tool, called RTC Quick Planner which allows teams to edit plans in a fast and fluid manner. RTC Quick Planner loads plans much faster, as it only loads only a few screens of the plan at a time and then loads additional plan items as you page through the plan and ask to display them. Developers, Scrum Masters and other users who need to load plans quickly and create new work items dynamically, should be encouraged to use RTC Quick Planner for their daily plan operations.

Slow running operations

Synchronizing attributes after a large amount of change in attributes for a work item

There are two ways for synchronizing work item attributes. First, using the RTC Eclipse client, run a query and select a set of work items then right click the type column and select Synchronize Attributes. This process would only affect the selected work items and response time is not a concern unless the client selected thousands of work items. The second method is to trigger the sync from the process configuration editor by the "check attributes usages in repository" action. This process could take a long time as it sync all work items created with all work item types within the same project area. If there are thousands of work items in the project area with a large number of attributes changes, it can take some time to complete (a warning is presented to the user).

Loading a work item with a large amount of customized attributes

When there are over 100 custom attributes in a work item type, loading of a work item of that type can be slow from the client perspective but isn't known to drive server load.

Rational Quality Manager

Duplicating test plans with a large test hierarchy.

The impact of duplicating a test hierarchy depends on the number of items and their size, and whether you choose to copy referenced artifacts. A test plan might include multiple child test plans, each with their own test cases and test scripts, resulting in potentially thousands of artifacts, each of which could reference a large amount of content storage. While you can count the number of objects ahead of time, you cannot determine the overall memory size of the selected hierarchy. Because the duplication occurs in a single transaction, it can require a high amount of memory to complete.

See Copying huge Test Plan brings server down with Out of Memory errors

Best Practices for Duplicating test plans with a large test hierarchy.

A best practice is to not do a deep copy and only copy references to test cases, test scripts, etc. Should a deep copy be needed, break down the overall hierarchy duplication into smaller subsets. If that is not possible, and you experience memory issues, you could pause the duplication when available system memory gets below some defined threshold, retry some number of times, then fail or rollback if memory remains below the threshold.

An even better best practice is to move away from duplication altogether in support of reuse by clone and own. Instead, transition to use of Configuration management in the QM application.

Bulk archiving and/or deletion of test results at end of release

If you select more than 10000 artifacts to archive or delete in bulk, the operation can take a long time and might time out. See Web browser-based artifact deletes greater than around 10,000 items will fail to execute.

Best Practices for Bulk archiving and/or deletion of test results at end of release

Either select a page of results at a time or ensure that less than 10000 artifacts are included in the result set, that is, work with smaller sets of assets.

Stream Creation

Creation of a stream requires cloning of internal database structures. As a local configuration gets bigger and bigger, stream creation will take longer and drive load on the system, especially if multiple streams are created at once.

This scenario needs further analysis to confirm it can be expensive and to characterize further. See work item TBD.

Jazz Reporting Services (all reporting technologies)

BIRT reports based on live data

Reports on live data run more slowly than those using the Data Warehouse or LQE, which are optimized for reporting. In addition, custom BIRT reports can be inefficient in their construction or pull large volumes of data, increasing load.

Best Practices for BIRT reports based on live data

Where possible, replace BIRT reports with reports constructed in Report Builder against data from the Data Warehouse or LQE. Ensure that you extensively test all BIRT reports before deploying them into production; tests should include a “worst-case scenario” that causes the most data to be fetched.

Reports that include more results or attributes than necessary

Regardless of data source or technology, scope your reports to include only the data required by the report consumers. Reports that include more results or attributes take longer to process and render.

Best Practices for Reports that include more results or attributes than necessary

If you are writing BIRT reports, narrow the result set as early as possible in the query processing. Retrieving large amounts of data, then applying filters or joins, can increase the run time and the memory demands. The same applies for any queries that you edit or write in Report Builder (in the Advanced section); move triple clauses up or down in the query based on performance during testing. Optimize and tailor the queries you run most often.

If you require reports that return a large number of results, run them during off-hours or when server usage is light.

Reports with large result sets on frequently-accessed dashboards

Widgets with multiple pages of results place additional demand on the server, while dashboard users are unlikely to click through all of those results. The same applies to widgets based on BIRT reports or those returning a large number of results.

Best Practices for Reports with large result sets on frequently-accessed dashboards

Ensure your dashboard widgets provide a relatively small result set. For dashboards with high or frequent use, avoid including widgets based on BIRT reports or those returning a large number of results; alternatively, put them on low use tabs.

Data aggregation and trend reports with large time intervals

Data aggregation reports (such as the BIRT RTC Timesheet and RQM Test Execution reports) and some time series or trend reports can take longer to run, depending on the amount of historical data in the data source. Using larger time intervals increases the load on the server, although repository size can also be a factor.

Best Practices for Data aggregation and trend reports with large time intervals

Limit the time interval for trend reports, and ensure you test the performance in a “worst-case scenario”.

DCC jobs requiring high storage and processing power

Most DCC jobs can run at regular intervals, obtaining a delta from the previous run. However, a few DCC jobs involve a larger amount of data, and place higher demands on storage and processing power.
  • Activity Fact Details (Activity History)
  • Build Fact Details (Build History)
  • File Fact Details (File History)
  • Project Management Fact Details (Project Management History)
  • Quality Management Fact Details (Quality Management History)
  • Requirement Management Fact Details (Requirement Management History)
  • Request Management Fact Details (Request Management History)
  • Task Fact Details (Task Fact Details)
  • Jazz Foundation Services - Statistics

Best Practices for DCC jobs requiring high storage and processing power

Schedule the identified jobs to run during off-hours or when server load is light. (Note: Job names listed are based on v6.0.2; where different, the names for earlier releases is in parentheses).

LQE index maintenance

Backup, compaction, re-index and addition or removal of data sources can drive load on LQE.

Best Practices for LQE index maintenance

Schedule these operations during off-hours or when server usage is light.

High-volume and highly complex queries

High-volume and highly complex queries can put a heavy load on the data source. As indicated above, ensure your reports return only what is necessary for the report consumer.

Report Builder provides an Advanced section where users can edit the queries generated by Report Builder, or write custom SQL (DW) or SPARQL (LQE) queries for a report. Inexperienced users can easily write and run an inefficient or incorrect query that could cause the data source become unresponsive.

For LQE, you can set properties in LQE's Query Service, such as the result limit (default is 3000 results) and query timeout (default is 60 seconds). LQE limits SPARQL queries based on these settings. (Note that this does not apply to SPARQL queries against metadata.)

Best Practices for High-volume and highly complex queries

Restrict access to the custom query editing; Report Builder provides this option for each data source.

LQE also offers a simple Query interface in its Administration UI. You can use this interface to run sample queries to discover information and improve your queries. You can also copy SPARQL from the Advanced section in Report Builder into this interface and make small changes to debug issues. This UI can target different scopes or configurations, or all data. Be aware that queries run from this UI still impact the data source, and are subject to the LQE Query Service limits, as described above.

For more information on improving LQE performance, see Monitoring and managing the performance of Lifecycle Query Engine and Improving Lifecycle Query Engine performance.

Refreshing a data source in Report Builder

Refreshing a data source in Report Builder queries the data source for the latest metadata, and can increase demand on both LQE and RB servers and impact the performance of other running reports. This is especially true for LQE data sources, where most of metadata must be queried (whereas most of the data warehouse metadata is hard-coded in Report Builder). Many factors affect the length of a refresh: the number of project areas, the complexity of their data model (for example, a large number of enumeration values), and the amount of change.

If a refresh is in progress, when a user accesses the Report Builder UI, a message displays and the user must wait until the refresh completes.

Best Practices for Refreshing a data source in Report Builder

Report Builder refreshes the data sources when the server starts. It also runs a background job twice daily to automatically refresh all data sources. Configure the refresh to run at times of lighter load for your organization, using the following properties in the server/conf/rs/app.properties file:
  • metamodel.autorefresh.time=6\:00AM
  • metamodel.autorefresh.repeat.inminutes=720

Administrators can refresh individual data sources on demand; be aware of the potential impact and refresh when it is less likely to affect other report users.

Global Configuration

Creating streams in a global configuration hierarchy

When you create a new stream for a global configuration, it may also generate new streams for the DNG, RQM, and DM contributions, and may need to traverse the global configuration hierarchy as well to create global streams from global baselines. The time to generate the streams depends on the complexity of the configuration hierarchy, the number of local application (DNG, RQM, and DM) baselines in the hierarchy, and number of versioned artifacts in each.

Because the local application (DNG, RQM, DM) creates its streams, most of the demand is placed on those servers. If there are a large number of baselines to create in the application servers, or if the GCM hierarchy is very deep or complex, you might want to create the new streams during a period of light usage.

This scenario needs further analysis to characterize further and establish best practices to limit impact. See work item 389494.

Creating a baseline staging stream from a global stream hierarchy

When you create a baseline staging stream (BSS) for a global configuration hierarchy, you also create a new baseline staging stream for each global stream in the hierarchy. The time to do this depends on the number of global configurations in the hierarchy, how deeply nested they are, and the number of custom properties and links they have.

This scenario needs further analysis to confirm it can be expensive and to characterize further. See work item 389497.

Creating a global baseline (or committing a baseline staging stream) from a global stream hierarchy

When you create a global baseline (or commit the baseline staging stream) for a global configuration hierarchy, it requests baselines for each local configuration from the contributing application (DNG, RQM, or DM). It also traverses the global configuration hierarchy, committing each included baseline staging stream and requesting local configuration baselines for those streams as well.

Similar to creating streams from the global baseline, much of the processing is done by the contributing application servers (DNG, RQM, and DM). If there are many local configurations, consider creating the baseline when usage of those servers is light.

This scenario needs further analysis to characterize further and establish a best practices to limit impact. See work item 389495.


Related topics: Deployment web home, Deployment web home

External links:

Additional contributors: TWikiUser, TWikiUser

Edit | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r7 | r4 < r3 < r2 < r1 | More topic actions...
 
This site is powered by the TWiki collaboration platformCopyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.