Best Practices for Configuring LQE For Performance and Scalability

Hardware Recommendations

Detailed hardware recommendations for Lifecycle Query Engine are contained within IBM Documentation topic: Configuring Lifecycle Query Engine with Jena or Link Index Provider with Jena for improving performance and scalability.
>LQE rs have documented their system requirements. There is also an LQE rs performance guide to further help you define your hardware needs.

Validating or re-indexing a large data source

Validating all the TRS feeds that are registered in the Lifecycle Query Engine or LDX index can increase your confidence in the quality of your reports and help you troubleshoot missing, stale, or extra artifacts. TRS Validation works by itself, without accessing Lifecycle Query Engine or LDX. It compares only the actual and expected content in TRS feed. Reference Troubleshoot Configuration-aware Reporting for further discussion on when this appropriate.

Note:

• Do not run more than one TRS validation operation at a time for a given TRS.
• If the application server is stopped during a TRS Validation operation, then, when the server is restarted, the TRS validation is not automatically restarted and it is in an unknown state. The validation must be started again and the admin user must set the option to clear the cache when starting the new validation.

There are occasions where the entire data source needs to be recreated. For example, after DOORS Next performs a full rebase operation followed by reindex of the DOORS Next Resources data source in LQE. The next section addresses how to optimize this use case, for large systems.

Recommendations for optimizing throughput during a large LQE data source reindex or validate

When validating TRS Feeds, we recommend 1GB+ of free temp disk space for each set of 1 million TRS entries

Large deployments of the Engineering Lifecycle Management will create a large amount of large selections. Selections are stored in the relational database in both versions of LQE (Jena-based and "rs"). We recommend that you amend the Selections Relational Database Batch Size: to at least 10000 (Default 425. Max: 50,000). The impact of this will be to make larger, and therefore faster updates of the large selections at all times. This is especially important when there are large changes to the data source, such as a complete reindex of the DOORS Next Resources data source.
If there are a large amount of changes in the data source, such as via archiving, then also amend Deletion Task Query Limit. This setting can be up to a maximum of the Relational Database Batch Size setting.

If the reindex operation is slow on Windows, consider using LQE direct I/O mode for reindexing. For more information, see Initial indexing is slower than expected when LQE/LDX is running on a Microsoft Windows server.

During reindex LQE will be trying to load the TRS data from DOORS Next; each request will briefly use a CPU core of the DOORS Next server. To speed up the reindex (at the cost of load on DOORS Next) increase the number of parallel requests by LQE to DOORS Next by setting Number of threads to fetch artifacts when adding or reindexing this data source on the DOORS Next Resources data source.
The default is 2. A starting point for increasing this would be two less than the number of CPU cores on the DOORS Next server. Monitor the load on DOORS Next during the reindex and increase/decrease this setting as appropriate.

For reindexing TRS Feeds:

• Batch Queue Size: 5000 (default 500)
  
• TRS HTTP Cache Max Entries: 5000 (default 100)
  
• Write Queue Size:500 (default 25)
  
• Process queue size: 500 (maximum is 1000)
  
• Batch Size for Writes: 100000 (default 2500)
  
• Batches Between Status Writes: 1
  
• Selections Relational Database Batch Size: 25000.
  
• Deletion Task Query Limit : 10000
  

If the above settings are adopted for a reindex of a DOORS Next data source, then to avoid the application becoming a bottleneck, especially on a production system, ensure to size the configuration cache to match the number of configurations:

a. Execute the SQL SELECT COUNT (*) FROM RMUSER.VVCMODEL_CONFIGURATION where type in (0,1,2) and archived='0';
b. Set the following setting in the jvm.options file as a Java argument to match the "Number of Active (nonarchived) configurations in the Repository" query

# JVM server parameters
-Dcom.ibm.rdm.configcache.size={# of configs from query}

Related topics: Deployment web home, Deployment web home

External links:

• IBM

Additional contributors:

• KrzysztofKazmierczyk, IanBarnard