Validating TRS feeds and the LQE index

Validate all the tracked resource set (TRS) feeds registered in the Lifecycle Query Engine (LQE) index when Report Builder reports don’t show the artifacts that you expect. Validating these items can help you diagnose and troubleshoot problems and avoid reindexing data sources.

About this task

You might validate all the TRS feeds registered in the LQE index in these situations:
  • A report is missing artifacts.
  • A report includes outdated (stale) artifacts.
  • A report shows the same artifact more than once, possibly with different attribute values or with its old and new name.
  • An application view and a corresponding report in Jazz® Reporting Service show a different number of artifacts.
  • A report includes deleted artifacts.
  • LQE shows skipped resources.
Validating all the TRS feeds that are registered in the LQE index can increase your confidence in the quality of your reports and help you troubleshoot missing, stale, or extra artifacts:
  • Validating the LQE index compares the current TRS feed from an IBM Engineering Lifecycle Management (ELM) application with the LQE index.

Validate TRS feeds when server usage is low, such as during a scheduled maintenance window.

The steps to validate TRS feeds are different across ELM applications. Complete the steps in this topic for the application whose feed you want to validate.
To learn more about validation before you do it, see the following resources:

For a description of TRS feeds in ELM, see Adding Tracked Resource Set feeds for lifecycle management applications.

Jazz Team Server, Engineering Workflow Management, Engineering Requirements Management DOORS Next, Engineering Test Management, and Global Configuration Management

Validate TRS feeds on the TRS Feed Validation page in the administration section of the application.

Before you begin

For Engineering Test Management:
  • When you run validation, you must have at least 1 GB of free disk space for each set of 1 million test artifacts in your repository. This total includes planning and execution artifacts, such as test plans, test cases, test scripts, test script steps, test execution records, test execution results, test execution result steps, test suites, test suite execution records, and test suite execution results.

About this task

Here are some examples of TRS feeds:
  • JTS User Resources (TRS 2.0): This feed contains information about Jazz Team Server (JTS) users. If a report about system users isn't showing the expected results, and you confirm that 10 users are registered in the repository, the validation task checks whether the TRS feed has 10 distinct references to users.
  • EWM Process Resources (TRS 2.0): If a report doesn’t show the project areas, team areas, or iterations that you expect, the validation task can help you determine whether they are missing from this TRS feed. If a user runs a report that doesn't include artifacts from a project that the user belongs to, TRS feed validation can fix the permission problem. After the validation, the report shows all the expected data.
  • GCM Resources (TRS 2.0): This feed contains information about changes to global configurations, for example, when they were created, updated, or deleted.
Tip:
If you are using ETM, see the note after this procedure.

Procedure

  1. Open the application administration page for the TRS feed to validate:
    • Jazz Team Server: https://host:port/jts/admin
    • Engineering Workflow Management: https://host:port/ccm/admin
    • Engineering Requirements Management DOORS® Next: https://host:port/rm/admin
    • Engineering Test Management: https://host:port/qm/admin
    • Global Configuration Management (GCM): https://host:port/gc/admin
  2. On the toolbar, click TRS Feed Validation.
  3. Select a TRS feed and clickValidate > Validate now.

    Validation examines the changes in the TRS feed. The change logs are recorded and stored since the first validation or since the last time that incremental data was reset. The validation results show errors from all the saved change records.

    If you no longer need the logged information, you can discard the old data, and validate the TRS feed from scratch. To remove all saved change records, select the Reset incremental data check box. You might want to reset it if, for example, errors in the TRS feed are resolved and the LQE index was updated.

    If the validation finds discrepancies, you see a list of missing or extra artifacts. Click the links and examine the information for why the artifacts are missing or added to the TRS feed. If you can't find the cause of the problem, contact IBM Support.

    If the data in the repository matches the TRS feed but you still see discrepancies in reports after validating the feed, consider running a full validation. This operation takes longer because it examines every entry in the TRS feed.

    Alternatively, for ETM you can also go to https://host:port/qm/TRSValidator and complete the following steps.
    1. For a full comparison of the TRS feed artifacts with those in the repository, click compare > runCompare. When the comparison starts, you can check the progress or stop the process.
    2. To see the log of all validations that ran on the system, or any validation that it’s still in progress, click trsLogs.
    3. When the comparison ends, check the logs for messages, such as Items match exactly or Discrepancies were found.
    4. If discrepancies are found, they are automatically fixed if you set the Automatically resolve discrepancies option for the com.ibm.rqm.oslc.trs.validator.TRSValidationWatchdog property to true.

      If you didn't change the advanced property, you can start cleanup events at https://host:port/qm/TRSValidator by clicking compare > cleanDifferences.

    FOR ETM ONLY:

    You can use an alternate ETM validation tool to automatically fix any discrepancies. The tool requires an advanced property before you start the validation process.

    To set the property, go to the administration page at https://host:port/qm/admin and find the Advanced Properties section. Then, change the Automatically resolve discrepancies option for the com.ibm.rqm.oslc.trs.validator.TRSValidationWatchdog property to true.

What to do next

If the problem persists, ask an LQE administrator to validate the LQE index.

Lifecycle Query Engine

Validate the LQE index to compare it with a TRS feed from an ELM application. This task finds missing, stale (outdated), or extra artifacts.

Before you begin

  • If you upgrade from 6.0.6 or an earlier version, a migration task creates a new partition that is used for validation and shortens reindexing time. This process might take some time.
  • If you upgrade from any earlier version, add the following properties to the JazzInstallDir/server/conf/log4j.properties file to ensure that your validation logs use the latest format.
    # Validation Task
    log4j.logger.com.ibm.team.integration.lqe.lib.indexer.tasks.ValidationTask = info, lqeValidationTaskLog
    log4j.additivity.com.ibm.team.integration.lqe.lib.indexer.tasks.ValidationTask=false
    log4j.appender.lqeValidationTaskLog=org.apache.log4j.RollingFileAppender
    log4j.appender.lqeValidationTaskLog.MaxFileSize=10MB
    log4j.appender.lqeValidationTaskLog.MaxBackupIndex=5
    log4j.appender.lqeValidationTaskLog.File=logs/lqe-validation.log
    log4j.appender.lqeValidationTaskLog.layout=org.apache.log4j.PatternLayout
    log4j.appender.lqeValidationTaskLog.layout.ConversionPattern=%d{ISO8601} [%30t] %5p %-50.50c  - %m%n

About this task

ELM application administrators might ask you to validate the LQE index after they validate TRS feeds.

Consider the following points before you validate the LQE index:
  • Run this task when system load is low. The time required for validation depends on the size of the index.
  • Don’t validate all the TRS data sources at the same time. Validate large data sources on separate days.
  • Validate only one TRS feed for one data source (ELM application) at a time. For ELM applications that have more than one TRS feed, run a separate validation for each feed.
  • Administrators can run LQE validation on a regular basis to confirm whether the LQE data has been kept in sync with the TRS feed from the data source. For example, if you run weekly validation and resources of the same type are logged as missing, you should check the data-providing application. You might need to change or configure a setting in that application. If the validation task runs for a few months without reporting any issues, you can extend the validation schedule. For example, you can run the task monthly.

To stop a validation that is running, restart the LQE server.

Procedure

  1. Open the Data Sources page: https://host:port/lqe/web/admin/data-sources
  2. Click the name of the TRS feed to validate.
  3. Click Validate near the upper right of the page.
    Validate button on the data source page
  4. Optional: Choose when to validate the data source by selecting a date and time.
  5. Click Validate and follow the prompts.

    While LQE reads the TRS feed, you see this message: Waiting for a response from the data source. This step can take a while, depending on the number of resources in the feed. This is an expected behavior, and no action is needed.

  6. Optional: Schedule LQE validation activities on a regular basis.
    1. Click Edit Schedule in the Validation Schedule section.
    2. Select the Enable Schedule check box.
    3. Specify the time and days of the week when you want to run the validation.
    4. Save your changes.

    When you enable a validation schedule for a TRS feed and it is the next scheduled task, the scheduled validation time is displayed in the Scheduled Tasks column of the Data Sources page.

Results

When the validation finishes, the results are published on the Data Sources page, at https://host:port/lqe/web/admin/data-sources. Click the status link of a data source:
Data source status link
On the Data Source Status page, you can see how many resources were updated (because they were either missing or stale), and how many extra resources were removed.
Validation results on the data source status page
On the Overview tab, click Download Logs to retrieve all the LQE logs for this data source.
Download logs button
The validation log files are named lqe-validation-<encoded_data_source_URL>.log. The logs list resource discrepancies in the following categories:
  • Missing resources: These resources are included in the TRS feed but not in the LQE index. The validation process will fetch these resources again to resolve the discrepancy.
  • Stale resources: The resources in the TRS feed are newer than the resources in the LQE index. The validation process will fetch these resources again to resolve the discrepancy.
  • Extra resources: These resources are included in the LQE index, but not in the TRS feed. The validation process will delete these resources to resolve the discrepancy.

You can enable email notification for validation events. For details see Setting up email notification for Lifecycle Query Engine events.

If you still see discrepancies in the content of a report after validation, consider reindexing.

After you click Validate, no further changes are indexed for that data source. If you scheduled validation to run later, no changes will be processed from the moment you click the validation button, until the validation finishes.