(r6) DOORSNextTRSTroubleshooting < Deployment

Engineering Lifecycle Management Wiki - Deployment
Community information and contribution guidelines
~~EditAttach~~Printable
TWiki > Deployment Web > DeploymentTroubleshooting > DOORSNextTRSTroubleshooting
Revision 6 - 2025-07-17 - 13:08:57 - WillChatham
<div id="header-title" style="padding: 10px 15px; border-width:1px; border-style:solid; border-color:#FFD28C; background-image: url(<nop>https://jazz.net/wiki/pub/Deployment/WebPreferences/TLASE.jpg); background-size: cover; font-size:120%">
---+!! Engineering Requirements Management (ERM) DOORS Next TRS Validation  
%DKGRAY% Authors: Main.WilliamChatham, Main.PetruAcsinte, Main.YuriyYermakov, Main.IanGreen, Main.PaulEllis, Main.IanBarnard<br>
Build basis: V7.0.2, V7.0.3 and V7.1
%ENDCOLOR%</div></sticky>

<!-- Page contents top of page on right hand side in box -->
<sticky><div style="float:right; border-width:1px; border-style:solid; border-color:#DFDFDF; background-color:#F6F6F6; margin:0 0 15px 15px; padding: 0 15px 0 15px;">
%TOC{title="Page contents"}%
</div></sticky>

<sticky><div style="margin:15px;"></sticky>

The objective of page is to provide an overview of the TRS Validation including troubleshooting.<br> 

---+ How to run an RM TRS Validation:
---

Running a TRS Validation may take a long time to complete and will put additional load on the server. 
   * We recommend running a validation when server usage is low, such as during a scheduled maintenance window.
   * There is no way to stop a validation once it starts except by shutting down the RM server.
      * Note: The validation will not automatically resume when the server restarts, the validation must be started again after clearing the TRS cache (As its in an unknown state).
   * Validation results are based on the data from when the validation starts, so if users make changes these changes will not be picked up and validated.
      * This is why we recommend running during a scheduled maintenance, otherwise, inaccurate validation results can be reported.
      * Running a validation during working hours is best to run against baselines only as these are immutable so cannot be affected by changes.
  

*Note:* If not already set, the DOORS Next TRS Validation requires the Selections Validation disk cache directory advanced property needs to be set before validation, pointing to a directory on the application server. To set this property, go to *https://host:port/rm/admin* and find the Advanced Properties page and allocate a directory with sufficient storage. We recommend that the Selections Validation disk cache directory is placed on a different file system than the DOORS Next server to prevent issues if the disk becomes full. In order to calculate the disk space requirements needed for the Selections Validation disk cache directory, you must use the Required file system space estimate tab on the RM TRS Feed Diagnostics page.

---++ Determine the Selections Validation disk cache directory size steps:

   1. Conduct a full rebase see [[https://www.ibm.com/support/pages/node/5695407][Perform a Rebase and Reindex for IBM Engineering Requirements Management DOORS Next]]
   1. When the rebase is complete, navigate to the TRS Feed Diagnostics page and select the Required file system space estimate tab to show the estimated directory size.<br>

     <img src="%ATTACHURLPATH%/TRS_Required_file_system_space_estimate.png" alt="TRS_Required_file_system_space_estimate.png" width="800" height="400" /><br><br>

Due to the time involved running a TRS Rebase on production, we advise that you run the full RM TRS Rebase on a comparable test system.  If a TRS Rebase is not possible, then you must monitor the disk space consumption during the validation operation and adjust storage allocation as necessary.<br><br>

---++ Procedure to run an RM TRS Validation:

Open the Engineering Requirements Management DOORS Next application administration page: https://host:port/rm/admin. On the toolbar, click *TRS Feed Diagnostics*.
In the Validate dialog box, click *Validate artifacts and configurations*.<br><br>

     <img src="%ATTACHURLPATH%/TRS_Validation_options.png" alt="TRS_Validation_options.png" width="600" height="700" /><br><br>

---+++ Validation options explained:
---

---++++ Validate configurations option:
---
Available when the application supports validation of configurations in the repository. Use this option to check that configurations have the correct members, and that those members are present in the TRS.

When *Validate all configurations* option is selected, the following happens:
TRS Validation performs a more thorough analysis of all configurations in the repository. Configuration membership is verified to agree with current state in the repository, and each configuration member is checked to ensure that it is present in the TRS.
*Note:* This option must be used consistently whenever a validation is done. Otherwise, the validation cache may become stale and validation discrepancies will result. Those discrepancies can be resolved by using the Clear cached validation data option.

When *Validate specific configurations* option is selected, validation focuses only on the chosen configurations to validate, rather than all configurations and all resources in the TRS. It may be quicker to validate a few configurations than all configurations. Further, when you do not care about the validity of other configurations. You can select any combination of local and global configurations, and validation will check the union of all local configurations, which are contributed to each chosen global configuration. 

*Note:* Notice that only configurations from the current application are checked. The membership of each local configuration is checked against the repository, and the TRS is checked to ensure that all configuration members are present.<br>

---++++ Validate artifacts option:
---
The Validate artifacts option can be used to verify that resources in the TRS can be fetched from the current application. Use this option if LQE is detecting 'skipped' resources. This option is mandatory when validating all configurations in the TRS. When validating specific configurations, only configurations and their members are validated.<br>

---++++ Resolve problems option:
---
Sometimes, applications might choose not to provide any repair capabilities in TRS Validation. So, this option is available only when application supports TRS feed repair. You can use the Resolve problems option in the Validate dialog box to automatically resolve discrepancies, if any. The checkbox is selected by default. This option is displayed only for the TRS feeds that support repair operation.

When there are unresolved discrepancies, you can see a list of
   * Missing or extra artifacts.
   * Unknown resource for Selections patch.
   * Wrong patch eTag.
   * Unexpected artifact added by a patch, etc.

The validation results are appended to the existing validation summary. You can view the number of repaired, missing, and extra artifacts in the feed. These artifacts are listed in separate sections.
   1. If all issues are resolved, then the validation result shows a green check mark.
   1. If there are discrepancies found that were fixed, then the validation result shows a Yellow Triangle.
   1. If there are discrepancies to fix, then the validation result shows a red cross. 

If you see discrepancies in reports after you validate the feed, consider running a full validation. This operation takes longer because it examines every entry in the TRS feed, discards all caches and starts with the beginning. The artifacts that cannot be repaired are listed as missing or extra artifacts. If you still see discrepancies, even after running the full validation, contact IBM Support.<br>

---++++ Clear cached validation data option:
---
When you are validating TRS feed, selecting this option, erases cached data that is recorded during previous validations that pertains to the current validation. Cached data helps to improve TRS Validation performance.

*Tip:* As you can validate either the TRS (with or without configurations) or specific configurations, it is important to use the same options each time. When validating specific configurations, options do not matter for the correctness of the next TRS validation.

Use this option in the following cases:
   1. If the application server is restarted while a validation as running as it is in an unknown state.
   1. After the TRS has been rebased or truncated, either when done manually, or as part of a scheduled periodic task.
   1. Following a validation run with incompatible options, such as, after you accidentally cleared the Validate configurations checkbox but still want to keep validating configurations. If you suspect that previous validations reported incorrect failures due to inconsistent validation options, such as when you tried validating a configuration with an outdated configuration cache entry. (This might happen when you stop validating all configurations when you are validating TRS and then decide to validate specific.)
   1. Selections Validation disk cache directory location change.<br><br>

---++ Throttling the TRS validation

TRS validation of the TRS providers, such as DOORS Next, or Engineering Test Management (ETM), is expected to be run <i>as frequently as required</i>.  If the validation is regularly resolving problems when run monthly, or if you are experiencing user reported incidents that match the "Why Validate?" section above, then you need to validate more frequently, for example, weekly.

Whether to use throttling depends very much on your environment.  Here are considerations, for evaluating whether you need to throttle:

   * How many TRS providers do you need to validate within your IBM ELM ecosystem...at the same time?
   * How many CPU do the machines running the TRS providers have?
   * How many IBM HTTP Servers (IHS) do you have/How many IHS connections are available?
   * What other off-hours automation are you running?  This includes custom, in-house as well as say DCC jobs.
   * Active Services is showing that each TRS validation thread is taking many minutes/hours for each thread

If, for example, you have multiple large ELM repositories of DOORS Next and ETM on the same shared resources, then you need to pay attention to the amount of time a validation takes/the overall CPU on the application server used and also if other users running their own scripts are complaining of longer durations to complete their scripts.

To throttle TRS validation feeds, you should:

   * Login to https://host:port/{application}/admin
   * Find <i> Maximum threads for resource validation </i>
   * Find <i> TRS Validation thread limit for repository queries </i>
   * Specify 25% of your available CPU for both these settings.

Note: the default for both values is 0, which equates to 100% of your CPU.  The 25% recommendation is a starting point.  Reduced CPU allocation could result in a longer time to complete.  However, if the number of TRS validation threads was causing issues, then the overall time may well improve when throttled.  [[https://www.researchgate.net/publication/226432744_Little%27s_Law][Little's Law]] has been encountered when all maintenance jobs are scheduled off-peak for users, creating a new peak for maintenance.

---++ When not to run validation

The Tracked Resource Set can become inconsistent in certain circumstances when configuration restores and archives are performed whilst the server is restarted.
If there is a server restart during archiving or restoration of configurations, then check the logs and Administrator GUI to ensure that they processed correctly.  If problems are suspected, then you must run TRS Validate to complete the compensating actions that propulate/remove the data from the TRS.

You should not validate during archive/restore, as you will be making changes that could cause the validate to report false positives, or fix (reverse) the action the restore/archive was performing.
To know whether you can validate pr not, the administrator can check the current status via:

   * The rm/admin -> Debug -> Manage Tracked Resource Set page where excluded containers will be listed in the watermark
   * Check the rmTRS.log, to see if compensating actions are being reported in the log.  You will need to set !TrsDeltaMonitorTask logger to DEBUG.
   * Run <i>select * form ARCHIVEMANAGER_AM_REQUEST</i>, only validate if this returns 0 rows

Note: if you run validation against specific configurations, then ensure you do not archive/restore from the same configurations.  If the data to be archived/restored, and the data to be validated are independent (not related) then there is no issue running both concurrently.

---+ Reading Validation results:
---
The results of each validation is stored in the GUI (By default, results are retained for 28 days) and the summary confirms how many records, artifacts and selections were validated and what failed validation and what was resolved. The validation options you selected are also recorded. Example below:<br><br>

     <img src="%ATTACHURLPATH%/Validation_results.png" alt="Validation_results.png" width="1374" height="425" /><br />

So in the example above:
   * Out of *4788* Artifacts validated *4786* were successful and *2* failed validation and those *2* were resolved.
   * Out of *27* Selections validated *13* were successful and *14* failed validation and could not be resolved so will need to be reviewed.<br><br>


---+ Troubleshooting:
---

---++ Logging:

The TRS Validation logs out to the *rm.log* which can roll over very quickly on a busy server so its best to have the validation log out to its own log (*TRS-Validation.log*). You can do this by adding the following to the *log4j2.xml* located here: *[installDir]/server/conf/rm/*.  Log4j2.xml changes are dynamically read, so there is no need to restart the server.

In the *Appenders* section, define a new appender by introducing:

<verbatim>
<RollingFile name="rmTRSValidation" filename="${dir}/${app}TRS-Validation.log" filePattern="${dir}/${app}TRS-Validation-%i.log">
	<PatternLayout pattern="${pattern}"/>
	<Policies>
		<SizeBasedTriggeringPolicy size="10 MB"/>
	</Policies>
	<DefaultRolloverStrategy max="5"/>
</RollingFile>
</verbatim>

In the *Loggers* section, include the following three loggers:

<verbatim>
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.validation" additivity="false" level="INFO">
	<AppenderRef ref="rmTRSValidation"/>
</Logger>
<Logger name="com.ibm.team.repository.service.internal.ReportsDiagnosticsRestService" additivity="false" level="INFO">
	<AppenderRef ref="rmTRSValidation"/>
</Logger>
<Logger name="com.ibm.team.repository.service.internal.trs" additivity="false" level="INFO">
	<AppenderRef ref="rmTRSValidation"/>
</Logger>
</verbatim>
<br><br><br>


---++ TRS Validation Results Downloader:
IBM Support can provide  the *TRS Validation Results Downloader* tool, which allows an Administrator the ability to download the results of the validation as these results can be quite large.

Steps to run the *TRS Validation Results Downloader* tool below:
   1. If not already installed follow the steps in the provided read me file to install the tool.
   1. To run the tool navigate to rm/admin and open the Debug tab followed by selecting L3 Tools.
   1. Select *TRS Validation Results Downloader*
   1. First input nothing into the fields, just press Download
   1. This will provide the list of previous validations and includes a UUID Token for each validation. 
   1. Copy the UUID Token for the TRS Validation you wish to check, e.g. "_5VO0kACVEe-xSu7KysdNHg"
   1. Paste this UUID Token into the Token field or leave blank if you wish to download all the available validations results. 
   1. For the location select a folder on the RM server where the results can be written to  e.g. C:\IBM\JazzTeamServer_7.0.2SR1\server\Validation
   1. Click on *Download*
   1. The results will be written to the C:\IBM\JazzTeamServer_7.0.2SR1\server\Validation folder and a separate folder is created for each validation denoted by the Token UUID.

The results break down as follows:

   * summary.txt : A Summary of the Validation results
   * extra.txt : Lists any extra resources reported in the validation.
   * missing.txt : Lists any missing resources reported in the validation.
   * failures.txt : Lists any failures reported in the validation.
   * repair.txt : Lists any repairs made during the validation.
   * selectionMissing.txt : Lists any missing selection resources reported in the validation.
   * selectionExtra.txt : Lists any extra selection resources reported in the validation.
   * selectionRepair.txt : Lists any Selection repairs made during the validation.
<br>
What to do with this output:<br>
In the location you specified above, for example: <i>C:\IBM\JazzTeamServer_7.0.2SR1\server\Validation</i> , go to that directory and see which of the above files were written.
<br>
IF there are repair or failure files repair.txt or selectionRepair.txt or failures.txt, are present and contain entries (not 0Kb) -> raise a case with IBM Support.
<br>
IF the files present are any of: missing.txt, extra.txt, selectionMissing.txt, selectionExtra.txt, then these are all false positives.<br>

<br><br><br>


---++ Known validation errors:
<br><br><br>
---+++ Selections URLs repeating in successive validations
---

*Possible causes*
   * Pre-iFix025: [[https://www.ibm.com/mysupport/aCI3p000000TX44][PH56481 Defect 145950 DOORS Next TRS validation cannot repair selections when TRS is running in event-based mode]] means repair of selections doesn't always work, so a selection which should have been repaired in one validation is also present in subsequent validations. this issue was fixed in *V7.0.2 ifix025* - see the workaround below

   * Post-iFix025: If you hit this issue create an IBM support case.

*Workaround: Log based TRS only*
   * None: Contact IBM Support.
<br>

*Workaround: Event based TRS only*
   * Pre-iFix025, Contact IBM Support and request the *L3 Selections Tool* (*v1.0.9* or greater) and Copy/Paste the repeated selection URL(s) from the validation report into the L3 TRS Selections Validate/Repair tool, using Reset+Repair.
<br>
To run the L3 Selections Tool (Provided by IBM Support) to workaround the issue (*Note this tool does not work in Log based TRS only Event based TRS*) do the following:
   1. Follow the install instructions in the read me.
   1. From the /rm/admin -> Debug page, select "L3 Services" and select TRS Selections Validation/Repair tool from the drop down list.
   1. Specify the URLs of the configurations to be validated or repaired in the text area. Copy and paste the ~Selections URL(s) on its own line. The URLs can be encoded or decoded.
   1. Select "Reset (non-recursive; recreates the selection resources for the configurations in scope regardless of the selection status)" and "Repair"
   * <img src="%ATTACHURLPATH%/TRS_Selections_Validator_tool_reset_repair.png" alt="TRS_Selections_Validator_tool_reset_repair.png" width="911" height="343" />


<br><br><br>
---+++ Unexpected status code in response: 400: Bad Request:
---

*Possible causes*
   * Possible problematic Artifact.
   * If the event states its “UnBlocked” it was removed from the ignore list which tries to process skipped resources again.
     
<img src="%ATTACHURLPATH%/Unexpected_status_code_in_response_400_Bad_Request.png" alt="Unexpected_status_code_in_response_400_Bad_Request.png" width="1224" height="167" />
 
*Workaround:* 
   * Possibly previously ignored Skipped resources unignored. Confirm if this was done.
   * New capability in the rm/admin -> Debug -> Manage Tracked Resource Set page we can use to check a Versions resource to see which configuration its part of and if it’s not part of a config it can be ignored. 

<img src="%ATTACHURLPATH%/Version_Resource_Usage.png" alt="Version_Resource_Usage.png" width="1227" height="214" />
<br><br><br>


---+++ Selections page ... cannot be retrieved. (Unexpected status code in response: 401: Unauthorized):
---

     <img src="%ATTACHURLPATH%/Selections_page_cannot_be_retrieved_Unexpected_status_code_in_response_401_Unauthorized.png" alt="Selections_page_cannot_be_retrieved_Unexpected_status_code_in_response_401_Unauthorized.png" width="1224" height="167" />

*Possible causes:*
   * Possible transient issue (temporary issue with the proxy Retry tends to work similar to 401s seen in LQE Skipped resources.
      * To retry do a specific validation on the affected Stream/Baseline. 
      * Make sure The Cache is cleared.
   * Possible issue with the URL (Percentage signs in the URL) Not a validation issue instead a Proxy issue, see if the below setting is set. See: [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.0.2?topic=server-configuring-reverse-proxy][Configuring IBM HTTP Server as a reverse proxy for WebSphere Application Server]].
      * *Optional:* Additional configuration for application ETL data collection jobs
      * If your application reporting ETL jobs are configured to use Jazz® Team Server (oAuth) authentication and there is an HTTP fronting server in the application topology, you must add an additional entry to the httpd.conf file.
      * Navigate to the installation directory for your IBM HTTP Server.
      * Open conf\httpd.conf in an editor.
      * In the Global section (Section 1) of the file, add this entry:
         * SetEnv websphere-nocanon 1
         * Restart your web server.
    |
*Workaround:*
   * Retry the validation for the specific configuration (make sure to clear the cache).
<br><br><br>


---+++ Too many Selections discrepancies. Validation process for the Selections resource interrupted
---
The message means that [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/571547][ PH59111 TRS validation is unable to repair selections with too many discrepancies]] a specific selection has more discrepancies, than the internal limit allows. [[https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/571547][ PH59111 TRS validation is unable to repair selections with too many discrepancies]] Was fixed in *V7.0.2 ifix028* and *V7.0.3 ifix003*.

<br><br>

*Workaround: Log based TRS only*
   * None: Contact IBM Support.
<br>

*Workaround:  Event based TRS only*
Contact IBM Support and request the *L3 Selections Tool* (*v1.0.9* or greater) and perform a *reset/repair* for the affected selection 

To run the L3 Selections Tool (Provided by IBM Support) to workaround the issue  (*Note this tool does not work in Log based TRS only Event based TRS*) do the following:
   1. Follow the install instructions in the read me.
   1. From the /rm/admin -> Debug page, select "L3 Services" and select TRS Selections Validation/Repair tool from the drop down list.
   1. Specify the URLs of the configurations to be validated or repaired in the text area. Copy and paste the ~Selections URL(s) on its own line. The URLs can be encoded or decoded.
   1. Select "Reset (non-recursive; recreates the selection resources for the configurations in scope regardless of the selection status)" and "Repair"
   * <img src="%ATTACHURLPATH%/TRS_Selections_Validator_tool_reset_repair.png" alt="TRS_Selections_Validator_tool_reset_repair.png" width="911" height="343" />
<br>
*Note:* this tool does not work with the new log-based TRS.
<br><br><br>


---+++ Missing and Extra resources reported in validation results (False positives):
---

*Root Cause:*
Issue fixed under: [[https://www.ibm.com/mysupport/aCI3p0000004NWn][Known Issue: DT363651: ERM TRS Validation reports false positive Extra and missing resources in the feed (WI 146949)]] . Available in ==V7.0.2 ifix031== and ==V7.0.3 ifix008==

*Workaround:*
To detect if you are seeing false positives contact IBM Support and request the *TRS Validation Results Downloader* tool (see here for more details:[[https://jazz.net/wiki/bin/view/Deployment/TRSValidation#TRS_Validation_Results_Downloade][TRS Validation Results Downloader]] ). This tool allows the Administrator to download the results of the validation and has the added benefit of being able to report if there are false positives. 

If the tool reports missing.txt or extra.txt along with the Summary.txt then these Extras and missing are false positives and can be ignored.)  So if there is no repaired.txt file in the results then they are all false positives.
<br><br>



---+++ TRS Validation results Summary reports Selection failures that are repaired:
---

This was raised under [[https://www.ibm.com/mysupport/s/defect/aCI3p0000004O2K/DT260140?language=en_US][DT260140: TRS Validation results Summary reports Selection failures that are repaired. (WI 572188)]].
The TRS Validation results Summary is incorrect and reports issues with “Selections failed validation” when they were actually repaired when reviewed in the full validation results.<br>
<br>
     <img src="%ATTACHURLPATH%/TRS_Validation_results_Summary_reports_Selection_failures_that_are_repaired.png" alt="TRS_Validation_results_Summary_reports_Selection_failures_that_are_repaired.png" width="682" height="628" />

Fixed in *Log based TRS Only* in ==V7.0.2 ifix031== and ==V7.0.3 ifix009==
<br><br>



---+++ Extras in selections in validation results:
---
*Variant 1* This was raised under [[https://www.ibm.com/mysupport/s/defect/aCI3p0000004O2K/dt363651?language=en_US][PH48108: EVENT BASED TRS SELECTIONS ARE NOT ALWAYS CORRECTLY UPDATED WHEN CHANGES ARE APPLIED TO THE SYSTEM (WI 144289)]]
(Closed as fixed in Log based TRS Only available in ==V7.0.3== and ==V7.0.2 ifix023==) Can result in Extra or missing selections.<br>
<br><br>

*Variant 2* Log based/Event based:  GUI shows: ID CRRRW7552E The specified artifact cannot be found in the current configuration. Unable to load
<br><br>
*Root cause:* Project area with no active components (All archived) returns an error in TRS Validation, GUI shows: ID CRRRW7552E The specified artifact cannot be found in the current configuration. Unable to load.  Technically, you can archive all configurations in a component as there is nothing to prevent a User from doing that. However there is no value in having such a component in a system. Issue fixed under: [[https://www.ibm.com/mysupport/aCI3p0000004NWn][Known Issue: DT269559: TRS Validation reports a component as Extra in the feed due to no active streams existing in a component (WI 146822)]] in ==V7.1==. 
<br><br>
     <img src="%ATTACHURLPATH%/TRS_Validation_failure_Extra_in_the_feed_due_to_no_active_streams_in_component_hover_shows_404_CRRRS08737W_There_is_no_usable_configuration_for_component.png" alt="TRS_Validation_failure_Extra_in_the_feed_due_to_no_active_streams_in_component_hover_shows_404_CRRRS08737W_There_is_no_usable_configuration_for_component.png" width="1464" height="369" />
<br><br>
*Workaround:*
The extra can either be ignored or else Unarchive the component and rerun the TRS Validation.
<br><br><br>
---+++ Missing from Selections in validation results:
---
This was raised under [[https://www.ibm.com/support/pages/apar/PH48108][PH48108: EVENT BASED TRS SELECTIONS ARE NOT ALWAYS CORRECTLY UPDATED WHEN CHANGES ARE APPLIED TO THE SYSTEM (WI 144289)]]
(Closed as fixed in Log based TRS Only available in ==V7.0.3== and ==V7.0.2 ifix023==) Can result in Extra or missing selections.<br>
<br><br><br>

---+++ Event with patch for selections resource unknown to validation in validation results:
---

     <img src="%ATTACHURLPATH%/Event_with_patch_for_Selections_resource_unknown_to_validation.png" alt="Event_with_patch_for_Selections_resource_unknown_to_validation.png" width="1104" height="169" />


Root cause unknown. 

*Workaround 1:* Perform a validation on the affected configuration, see steps below:

   1. For each URL you will need to identify the Stream name so in the example above edit the URL:  https://XXXXX/rm/configSelections/stream/_Xz9RIIRxEe6Gzpvqit1zkg as https://XXXXX/rm/cm/stream/_Xz9RIIRxEe6Gzpvqit1zkg and open the URL and take note of the stream name.
   1. In the TRS Validation page select "Validate specific configurations" and select the affected configurations.
   1. Check the "Clear cached validation data" option.
   1. Run the Validation.

NOTE: If you "Clear cached validation data" within the context of a specific configuration, then you only clear the cache data of that single configuration.  This will not clear the entire cache.

*Workaround 2:* For non log-based RM systems is to use the TRS Selections Validation/Repair tool to process multiple URIs concurrently.

To run the L3 Selections Tool (Provided by IBM Support) to workaround the issue (*Note this tool does not work in Log based TRS only Event based TRS*) do the following:
   1. Follow the install instructions in the read me.
   1. From the /rm/admin -> Debug page, select "L3 Services" and select TRS Selections Validation/Repair tool from the drop down list.
   1. Specify the URLs of the configurations to be validated or repaired in the text area. Copy and paste the ~Selections URL(s) on its own line. The URLs can be encoded or decoded.
   1. Select "Reset (non-recursive; recreates the selection resources for the configurations in scope regardless of the selection status)" and "Repair"
   * <img src="%ATTACHURLPATH%/TRS_Selections_Validator_tool_reset_repair.png" alt="TRS_Selections_Validator_tool_reset_repair.png" width="911" height="343" />
<br><br>

<br><br><br>


---+++ Addition of unexpected artifact by Selections patch:
---

*Root cause:* This is an issue where the problem was identified and fixed but is still reported as an issue in the Validation results, see this in the example screen print below:
<br><br>
     <img src="%ATTACHURLPATH%/Addition_of_unexpected_artifact_by_Selections_patch.png" alt="Addition_of_unexpected_artifact_by_Selections_patch.png" width="1169" height="189" />
<br><br>
This will be covered in defect, [[https://www.ibm.com/support/pages/apar/PH59596][PH59596: (WI572188 ) TRS Validation results Summary reports Selection failures that are repaired]]. Fixed in ==V7.0.2 ifix031== and ==V7.0.3 ifix009==.  
<br>

*Workaround:* Can be ignored if the Stream appears as repaired in the results.
<br><br><br>

---+++ Missing Deletion by selections patch in validation results:
---
     <img src="%ATTACHURLPATH%/Missing_deletion_by_Selections_patch.png" alt="Missing_deletion_by_Selections_patch.png" width="1150" height="143" />

Root cause unknown. 

*Note* We have seen this appear under the validation results but its benign and does not affect Customer reporting so can be ignored.

<br><br><br>

---+++ This stopwatch is already stopped reported in the rmTRS.log
---

If this message is encountered in the logs, contact IBM Support.  This is known issue [[https://www.ibm.com/mysupport/aCIKe000000fy3C][DT387232 - IllegalstateException: This stopwatch is already stopped in log-based mode]]. Fixed in ==V7.0.2 ifix030== and ==V7.0.3 ifix007==.

<verbatim>
 2024-07-03T18:28:17,259+0200 [TID: 78881AB9][lqe_user][Default Executor-thread-1660135 @@ 18:28 <unauthenticated> <LQE/1.0@1.1.1.2> /rm/configSelections/baseline/_-zbafdjhkdszO9PGMcasjA/query] ERROR com.ibm.rdm.fronting.server.services.trs           - Error processing HTTP request

	com.ibm.rdm.fronting.server.services.trs.ConfigSelectionsRestService@93035083

	This stopwatch is already stopped.
...
java.lang.IllegalStateException: This stopwatch is already stopped.
</verbatim>

If encountered pre *V7.0.2 ifix030*, then the server will need to be restarted and any selection/baseline republished.
<br><br><br>
---+++ !CRJAZ3160E The validation was canceled since XXX problems were encountered. Only 100000 can be processed at once:
---
The default limit for the number of problems the validation can fix is *100,000*.  If the Validation finds more than *100,000* problems it will fail with the following message in the logs:

<verbatim>
2023-11-27T15:59:32,056+0200 [TID: XXXXX][][ TRS validation thread 0]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task
com.ibm.team.repository.common.TeamRepositoryException: CRJAZ3160E The validation was canceled since 131172 problems were encountered. Only 100000 can be processed at once. The TRS feed may be damaged and should be rebuilt in its owning application.
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils$ValidationProcessor.validateTrsUris(InternalTrsValidationUtils.java:510) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils.validateTrsUris(InternalTrsValidationUtils.java:368) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService.lambda$0(InternalTrsValidationUtilsService.java:269) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidation.validate(InternalTrsValidation.java:64) ~[repository_service.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]
	at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at com.sun.proxy.$Proxy144.validate(Unknown Source) ~[?:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService.lambda$1(InternalTrsValidationUtilsService.java:285) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService$4.run(InternalTrsValidationUtilsService.java:571) [bundleFile:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) [?:1.8.0]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:1.8.0]
	at java.lang.Thread.run(Thread.java:825) [?:2.9 (04-17-2023)]
</verbatim><br>

The *100,000* limit can be increased. by following the procedure below:
   1. Under the rm/admin advanced properties set *Maximum TRS errors to display* to greater than the amount detailed in the error, if the number of errors are above *500,000* please contact IBM support for assistance.
   1. Under the rm/admin advanced properties set *TRS validation repair phase checks for false positives* to *false* (if there are 2 properties with the same name, set them both to false).  
   1. Rerun the validation.
   1. When it completes navigate to the rm/admin advanced properties and reset *TRS validation repair phase checks for false positives* back to *true*.<br><br>

<br><br><br>

---+++ !Failed deleting TRS Selection cache file:
---

This issue can occur if there are issues with the *Selections Validation disk cache directory*, if any files were manually deleted or manipulated the Validation will fail and the following error is logged.

<verbatim>
2023-12-11T10:25:19,149+0100 [TID: XXXXXX][][TRS validation thread 0]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task
com.ibm.team.repository.common.TeamRepositoryException: Failed deleting TRS Selection cache file /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache/selections_Selections.baseline.XXXXXXXXXXXXXXXXXX
	at com.ibm.team.repository.service.internal.trs.validation.selCache.TrsSelectionsCacheService.clearCachedSelections(TrsSelectionsCacheService.java:266) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.selCache.TrsSelectionsCacheService.clearAllCachedSelections(TrsSelectionsCacheService.java:284) ~[bundleFile:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]
	at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at com.sun.proxy.$Proxy145.clearAllCachedSelections(Unknown Source) ~[?:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils$ValidationProcessor.dropAllCaches(InternalTrsValidationUtils.java:584) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils$ValidationProcessorForStrings.loadCacheOrReadActiveMembers(InternalTrsValidationUtils.java:799) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils$ValidationProcessor.validateTrsUris(InternalTrsValidationUtils.java:441) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils.validateTrsUris(InternalTrsValidationUtils.java:368) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService.lambda$0(InternalTrsValidationUtilsService.java:269) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidation.validate(InternalTrsValidation.java:64) ~[repository_service.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]
	at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at com.sun.proxy.$Proxy144.validate(Unknown Source) ~[?:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService.lambda$1(InternalTrsValidationUtilsService.java:285) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService$4.run(InternalTrsValidationUtilsService.java:571) [bundleFile:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) [?:1.8.0]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:1.8.0]
	at java.lang.Thread.run(Thread.java:825) [?:2.9 (04-17-2023)]
Caused by: java.nio.file.NoSuchFileException: /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache/selections_Selections.baseline.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:98) ~[?:1.8.0]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:114) ~[?:1.8.0]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:119) ~[?:1.8.0]
	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:256) ~[?:1.8.0]
	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:115) ~[?:1.8.0]
	at java.nio.file.Files.delete(Files.java:1137) ~[?:1.8.0]
	at com.ibm.team.repository.service.internal.trs.validation.selCache.TrsSelectionsCacheService.clearCachedSelections(TrsSelectionsCacheService.java:263) ~[bundleFile:?]
	... 28 more
</verbatim><br>

We cache in memory the index file of the Selections cache directory. If the cache files are manually deleted while the server was running, then this issue can occur and can only be resolved with a DOORS Next server restart.<br><br>

<br><br><br>
---+++ Selections patch events in Change Log do not have expected beforeETag 1.ab1cd234.
---

The Validation result will show a Selection resource as flagged, when running against a specific configuration.

The result will state:

<verbatim>
https://servername/rm/configSelections/stream/_1a2BcD3eFghiLJ3XCVU6kA	Selections patch events in Change Log do not have expected beforeETag 1.ab1cd234.
</verbatim>

This may happen for two reasons:

   1. TRS validation runs were inconsistent.  See: [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.0.3?topic=administering-validating-trs-feeds-lqe-ldx-index][Validating TRS feeds and the LQE or LDX index]]
   2. TRS feed has a broken sequence of TRS patches for this particular Selections.

The recommendation in this case to validate this particular configuration using the Clear Cache option.

<br><br><br>
---+++ !Invalid TRS response.. Unexpected status code from GET.. 502: Bad Gateway
---

This Validation failure due to 502 Bad Gateway error can occur due to server load with connections.

<verbatim>
2023-11-24T15:01:03,804+0100 [TID: EB4D1F93][][       TRS validation thread 3]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task com.ibm.team.repository.service.internal.trs.feed.TrsFeedReadException: Invalid TRS response at <https://xxxx/rm/trs2ChangeLogs/_HIZhAXl0Ee6n7Ow9oWyCxw>;
...
Caused by: com.ibm.team.repository.service.internal.trs.feed.impl.TrsResponseException: Unexpected status code from GET to <https://xxxx/rm/trs2ChangeLogs/_HIZhAXl0Ee6n7Ow9oWyCxw>; 502: Bad Gateway
	at com.ibm.team.repository.service.internal.trs.feed.impl.Trs20FeedReaderService$TrsRdfResponse.create(Trs20FeedReaderService.java:676) ~[bundleFile:?]
</verbatim><br>

We recommend running the validation during a quiet period with less load on the servers and connections. So you can attempt a rerun during a quiet period. <br><br>

<br><br><br>

---+++ !Selections page https://rm/configSelections/stream/xxx/journal/xxx> Cannot be retrieved. (Unexpected status code in response: 410:Gone)
---

Validation failure can occur if the a stream selection has a problem retrieving a journal page, the selection is flagged as invalid by the validation but the validation cannot repair this.
<br>
     <img src="%ATTACHURLPATH%/410_Gone_Validation_error.png" alt="410_Gone_Validation_error.png" width="955" height="113" />
<br>

This identified as a defect and raised under [[https://www.ibm.com/support/pages/apar/PH58915][PH58915: OOTB TRS VALIDATION CANNOT REPAIR SELECTIONS WITH MISSING JOURNAL PAGES (WI 146583)]] . Fixed in ==V7.0.2 ifix028== and ==V7.0.3 ifix004==.


*Workaround: Log based TRS only*
   * None: Contact IBM Support.
<br>

*Workaround:  Event based TRS only*
   * Contact IBM Support and request the *L3 Selections Tool* (*v1.0.9* or greater) and perform a *reset/repair* for the affected selection (*Note:* this also requires defect [[https://www.ibm.com/support/pages/apar/PH58915][PH58915: OOTB TRS VALIDATION CANNOT REPAIR SELECTIONS WITH MISSING JOURNAL PAGES (WI 146583)]] to be applied. Issue is fixed in ==V7.0.2 ifix027==). 

To run the L3 Selections Tool (Provided by IBM Support) to workaround the issue do the following:
   1. Follow the install instructions in the read me.
   1. From the /rm/admin -> Debug page, select "L3 Services" and select TRS Selections Validation/Repair tool from the drop down list.
   1. Specify the URLs of the configurations to be validated or repaired in the text area. Copy and paste the ~Selections URL(s) on its own line. The URLs can be encoded or decoded.
   1. Select "Reset (non-recursive; recreates the selection resources for the configurations in scope regardless of the selection status)" and "Repair"
   * <img src="%ATTACHURLPATH%/TRS_Selections_Validator_tool_reset_repair.png" alt="TRS_Selections_Validator_tool_reset_repair.png" width="911" height="343" />
<br><br>

<br><br><br>


---+++ No space left on device
---
This message was reported after processing a large system, but can occur if the ValidationCache directory is not monitored/set with sufficient disk space.
This message is also encountered if the %TEMP% space is insufficient.

<verbatim>
com.ibm.team.repository.service.internal.trs.validation.resource.ResourceValidationException: Validating TRS change log in 'com.ibm.rdm.fronting.server.trs.validation': java.io.IOException: No space left on device
</verbatim>

One problem with this message, is the subsequent messages that you get when trying to run TRS Validate.  This occurs even if *Clear Configuration Cache* is selected:

<verbatim>
2023-12-30T05:41:32,854+0100 [TID: 1544B173][][       TRS validation thread 2]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task

com.ibm.team.repository.service.internal.trs.validation.resource.ResourceValidationException: Validating TRS change log in 'com.ibm.rdm.fronting.server.trs.validation': Failed deleting TRS Selection cache file /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache/selections_configSelections.baseline._k3sOYQp5Ee6yhpy2leP1DA._393oe6KwEe6pO7zRQeW-4g
	at com.ibm.team.repository.service.internal.trs.validation.resource.ResourceValidationExceptionCatcher.close(ResourceValidationExceptionCatcher.java:120) ~[bundleFile:?]

</verbatim>

*Root cause:* Issue occurs due to a known issue: [[https://www.ibm.com/mysupport/aCIKe0000008Owo][DT379224:TRS Validation fails due to Error encountered Validating TRS change log in 'com.ibm.rdm.fronting.server.trs.validation': Failed deleting TRS Selection cache file /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache/selections_configSelections.xxxxxxx(574208)]].
This often occurs if <i>Rebase<\i> has not been performed, or was performed some time ago.  The DOORS Next TRS Feed Diagnostics page also shows [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.0.2?topic=administering-validating-trs-feeds-lqe-ldx-index][Required file system space estimate tab on the TRS Feed Diagnostics page for DOORS Next resources]] that should be adhered to, to avoid this disk full scenario.

*Workaround for insufficient Validation Cache size:* Manually clear the cache (Delete the contents of the *Selections Validation disk cache directory* advanced property Location e.g. /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache folder) and restart the RM server. The subsequent attempt to run TRS validate will succeed.
<br>
If your investigation of disk space shows that there is sufficient space in the Validation Cache directory, then continue to evaluate the /tmp directory, or the temp space.  If this is deemed too small, then change the location that DOORS Next will use by applying the following workaround.

*Workaround for insufficient temp space*
   * cd [ELM_Install]/server
   * edit server.startup script. 
   * Change the following line to add your new path:
      * set JAVA_OPTS=%JAVA_OPTS% -Dcom.ibm.team.repository.tempDir="%TEMP%"

e.g. set JAVA_OPTS=%JAVA_OPTS% -Dcom.ibm.team.repository.tempDir=C:\IBM\JazzTeamServer_7.0.2SR1\test

<br><br><br>



---+++ Not found in Change Log
---

*Variant 1:*

This message was reported in the Logs after a TRS Validation:

<verbatim>
2024-02-20T02:02:117,117+0100 [TID: XXXXXXXX][][ TRS validation thread 1] WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task
com.ibm.team.repository.service.internal.trs.feed.TrsFeedReadException: Cutoff event(s) <[urn:_TOgnsOiVEeyOKKA0JZrFGA:1700273627117]> not found in Change Log
</verbatim>
<br>

*Root cause:* Issue occurs if the change log is truncated, or cache is stale, most common root cause occurs if a full rebase is run and the Validation is run without clearing the cache.
<br>

*Workaround:* The only resolution here is to clear the cache and run a fresh TRS Validation.
<br><br><br>


*Variant 2:*

This message was reported in the Logs after a TRS Validation:

<verbatim>
2024-02-23T14:02:52,873+0000 [TID: 3442C500][][ TRS validation thread 2] ERROR ervice.internal.trs.validation.ContentFormatHelper - Error encountered during validation
com.ibm.team.repository.service.internal.trs.feed.TrsFeedReadException: Cutoff event(s) <[urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:normal:E]> not found in Change Log. 
</verbatim>
<br>

*Root cause:* Issue occurs due to a known issue: [[https://www.ibm.com/mysupport/aCI3p000000LJ9C][DT270138:Incorrect storage exception handling could lead to TRS changelog truncation (146857)]]. This issue is fixed in ==V7.0.2 ifix029== and ==V7.0.3 ifix004==. 
<br>

*Identifying the issue:*

As well as the log entry above We also have a tool called the *TRS Feed helper* tool that can be used to check the event that failed in the validation: Example: "urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:normal:E"

This tool can be requested from IBM support, when run the tool returns the "Date of last processed changelog page" which was the time this event broke the change log: See example below:

<verbatim>
Starting execution at Wed Feb 28 14:19:22 UTC 2024

search(urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E)
TRS Types Helper v1.0.0
searchForTypeInfo(urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E)
Searching urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E ...
Wed Feb 28 14:19:27 UTC 2024 -> 100 changelog pages processed so far
Wed Feb 28 14:19:32 UTC 2024 -> 200 changelog pages processed so far
Wed Feb 28 14:19:37 UTC 2024 -> 300 changelog pages processed so far
Wed Feb 28 14:19:41 UTC 2024 -> 400 changelog pages processed so far
Wed Feb 28 14:19:47 UTC 2024 -> 500 changelog pages processed so far
Wed Feb 28 14:19:52 UTC 2024 -> 600 changelog pages processed so far
Wed Feb 28 14:19:56 UTC 2024 -> 700 changelog pages processed so far
Wed Feb 28 14:20:02 UTC 2024 -> 800 changelog pages processed so far
Wed Feb 28 14:20:06 UTC 2024 -> 900 changelog pages processed so far
Wed Feb 28 14:20:12 UTC 2024 -> 1000 changelog pages processed so far

METRICS

searchForEvent [
Total calls: 1
Avg: 64070
Min: 64070
Max: 64070]
RESULTS FOR urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E
NOT found, urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E
Wed Feb 28 14:20:27 UTC 2024, 1303 changelog pages processed
Date of current changelog page, Wed Feb 28 14:17:13 UTC 2024
Date of last processed changelog page, Tue Feb 13 16:23:42 UTC 2024
Done searching for information on urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E
</verbatim>

<br>
*Workaround:* The only resolution here is to run a full rebase and make sure you are running ==V7.0.2 ifix029== or ==V7.0.3 ifix004== to lessen the chance of hitting: [[https://www.ibm.com/mysupport/aCI3p000000LJ9C][DT270138:Incorrect storage exception handling could lead to TRS changelog truncation (146857)]] again. 
<br><br><br>



---+++ Failed to get the result of this validation

---

When opening the result of a TRS validation, the message below is encountered after 15 minutes:

<verbatim>
Failed to get the result of this valdation
A server error occurred: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body>
 <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> 
<p>Please contact the server administrator at you@your.address to inform them of the time this error occurred, and the actions you performed just before this error.</p> 
<p>More information about this error may be available in the server error log.</p> <hr> 
<address>IBM_HTTP_Server Server at servername.ibm.com Port 443</address> </body></html> - Error code 500
</verbatim>

*Workaround:* This only occurs when there is a very large result to return and the time exceeds the ServerIOTimeout set in the IBM HTTP Server (IHS) configuration.
To resolve this issue, you need to increase this ServerIOTimeout setting to a higher value.  In order to understand what value, you would need to track the Active Services page to see how long the transaction takes, then increase the ServerIOTimeout accordingly.


<br><br><br>
---+++ Validation terminates with <i>java.util.ConcurrentModificationException</i>
---

This scenario is likely to occur if running the TRS validation while there are other changes being made in the system.  It is advisable to run the TRS validations during off-peak hours to reduce the chance of encountering this issue.

The TRS Validation interface will report 

<verbatim>
The validation was canceled because of an error. For details, check the server log file.
</verbatim>

The logs will show:

<verbatim>
2024-01-28T21:48:20,567+0100 [TID: F6BBD78F][][       TRS validation thread 0]  INFO e.internal.trs.validation.TrsCachePopulationManger - Reading fetched TRS data for https://xxx/rm/trs2

2024-01-29T08:00:07,289+0100 [][TRS-Resource-Validation: validate-trs-resource <https://xxx/rm/configSelections/baseline/_K2BToH3lEe6CGdsfON46Tg>;]  INFO .validation.selCache.RegularFeedPropertiesSupplier - Selections Cache Storage Location for feed id "com.ibm.rdm.fronting.server.trs.validation" is /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache

2024-01-31T04:38:19,948+0100 [TID: F6BBD78F][][       TRS validation thread 0]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task

com.ibm.team.repository.service.internal.trs.validation.resource.ResourceValidationException: Repairing Selections resources in 'com.ibm.rdm.fronting.server.trs.validation': java.util.ConcurrentModificationException: java.util.ConcurrentModificationException: java.util.ConcurrentModificationException
</verbatim>

This is a known defect, [[https://www.ibm.com/support/pages/apar/PH58309][PH58309: (WI570317) TRS VALIDATION REPAIR MIGHT THROW CONCURRENTMODIFICATIONEXCEPTION]]. This issue was fixed in ==7.0.2 ifix028== and ==7.0.3 ifix006==. 

*Workaround:* You must re-run the validation.

<br><br><br>

---+++ Validation fails with an Out Of Memory(OOM) error
---

The TRS Validation could run into memory problems depending on the resources available and load on the server, the *rmTRS-Validation.log* will show:

<verbatim>
TRS validation thread 0]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task
java.lang.OutOfMemoryError: null
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:83) ~[?:1.8.0]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57) ~[?:1.8.0]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:437) ~[?:1.8.0]
	at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:609) ~[?:1.8.0]
	at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:688) ~[?:1.8.0]
	at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:746) ~[?:1.8.0]
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:171) ~[?:1.8.0]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:185) ~[?:1.8.0]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:244) ~[?:1.8.0]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:429) ~[?:1.8.0]
	at com.ibm.team.repository.service.internal.trs.validation.resource.ParallelStreamRunner.execute(ParallelStreamRunner.java:110) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.resource.ParallelStreamRunner.runForEach(ParallelStreamRunner.java:68) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.resource.TrsResourcesValidator.validateActiveMembers(TrsResourcesValidator.java:240) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.resource.TrsResourcesValidator.validateResources(TrsResourcesValidator.java:155) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils$ValidationProcessor.validateTrsUris(InternalTrsValidationUtils.java:504) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils.validateTrsUris(InternalTrsValidationUtils.java:380) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService.lambda$0(InternalTrsValidationUtilsService.java:300) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService$$Lambda$941/000000005F1E2BC0.call(Unknown Source) ~[?:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidation.validate(InternalTrsValidation.java:64) ~[repository_service.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]
	at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
	at com.sun.proxy.$Proxy143.validate(Unknown Source) ~[?:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService.lambda$1(InternalTrsValidationUtilsService.java:316) ~[bundleFile:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService$$Lambda$942/0000000059DD4850.call(Unknown Source) ~[?:?]
	at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService$4.run(InternalTrsValidationUtilsService.java:599) [bundleFile:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) [?:1.8.0]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:1.8.0]
	at java.lang.Thread.run(Thread.java:822) [?:2.9 (07-24-2020)]
Caused by: java.lang.OutOfMemoryError: Java heap space
</verbatim>
<br>

You may need to check the avaialble RAM and heap settings and make appropriate changes to accomadate the Valiation. 

If resources cannot be increased then there is another possible option to reduce the TRS Hybrid capacity settings in the RM advanced properties:

*TRS hybrid list capacity*
   * The number of elements for a data page.
   * -1 disables the hybrid list.
   * Limited to minimum 10,000).
   * Default is  "100,000"

*TRS hybrid set capacity*
   * The number of elements to store in memory before saving to disk.
   * -1 disables the hybrid set.
   * Limited to minimum 100,000.
   * Default is  "10,000,000"

To do this:
   * Browse to /rm/admin->Advanced Properties.
   * Search for the properties "TRS hybrid list capacity" and "TRS hybrid set capacity" and reduce the sizes.
      * e.g. *TRS hybrid list capacity* set to *10000* and *TRS hybrid set capacity* set to *1000000*.
   * Restart the server for these settings to take affect.
<br><br><br>


---+++ Validation results include message: *Read Timed Out*
---

Below the heading *Failed validaiton* there are entries which include the text *Read timed out*

These are recorded when the RM server doesn't respond in time to the validation request - this can happen when RM is very busy.

*Workaround:* Increase the RM socket timeout

To do this:
   * Browse to /rm/admin->Advanced Properties.
   * Below the heading TRS Validation, change setting Socket Timeout from 300000  to 900000
   * *Note:*  If you are running Log based TRS you must restart the server or else the TRS DMT will become suspended and only a restart will allow it to resume, fixed under defect: [[https://www.ibm.com/mysupport/s/defect/aCIKe000000XeeR/dt395747][DT395747 TRS DMT is suspended when modifying the TRS Validation Socket Timeout advanced property(147815)]] avaialble in V7.0.2 ifix032,  V7.0.3 ifix011 and V7.1 GA.



Then run the validation again with the exact same settings as the previous validation.


---++ Skipped resources showing in LQE on the DN Resources data source:

We have a separate Wiki to help you with skipped resources: [[HowToAnalyseAndFixDNSkippedResourcesInV7X][How to Analyse/Fix DOORS NEXT (DN) Skipped resources in V7.X]]

---++ IBM Support's additional must gather documents

In addition to skipped resources, common questions to IBM Support have been documented as tech notes.  Performance troubleshooting of LDX and LQE is required if the cause of the data issue can be attributed to the delay of getting the TRS data from the TRS provider to the TRS consumer.

   * [[https://supportcontent.ibm.com/support/pages/validated-link-doors-next-artifact-etm-test-case-not-visible-doors-next?check_logged_in=1][Validated By links are not visible in DOORS Next]]
   * [[https://www.ibm.com/support/pages/node/6128283][Mustgather: Investigating LQE or LDX performance problems]]


---+++++!! External links:          

   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.0.3?topic=administering-validating-trs-feeds-lqe-ldx-index][Validating TRS feeds and the LQE or LDX index]]
   * [[https://jazz.net/wiki/bin/view/Deployment/ConfigurationAwareReportingTroubleshooting][Troubleshooting configuration-aware reporting]]

---+++++!! Additional contributors: Main.BarryReilly

<sticky></div></sticky>
Attachments
Topic attachments
I	Attachment	Action	Size	Date	Who
png	410_Gone_Validation_error.png	manage	29.6 K	2023-12-22 - 14:18	WillChatham
png	Addition_of_unexpected_artifact_by_Selections_patch.png	manage	32.4 K	2024-02-20 - 13:54	WillChatham
png	Event_with_patch_for_Selections_resource_unknown_to_validation.png	manage	37.0 K	2024-01-29 - 19:08	WillChatham
png	Missing_deletion_by_Selections_patch.png	manage	23.4 K	2024-01-29 - 19:20	WillChatham
png	Selections_page_cannot_be_retrieved_Unexpected_status_code_in_response_401_Unauthorized.png	manage	32.1 K	2024-02-14 - 11:58	WillChatham
png	TRS_Required_file_system_space_estimate.png	manage	57.7 K	2023-11-27 - 10:13	WillChatham
png	TRS_Selections_Validator_tool_reset_repair.png	manage	65.4 K	2024-01-04 - 10:38	WillChatham
png	TRS_Validation_failure_Extra_in_the_feed_due_to_no_active_streams_in_component.png	manage	20.9 K	2024-02-27 - 12:09	WillChatham
png	TRS_Validation_failure_Extra_in_the_feed_due_to_no_active_streams_in_component_hover_shows_404_CRRRS08737W_There_is_no_usable_configuration_for_component.png	manage	42.4 K	2024-02-27 - 12:09	WillChatham
png	TRS_Validation_options.png	manage	37.3 K	2023-11-27 - 11:45	WillChatham
png	TRS_Validation_results_Summary_reports_Selection_failures_that_are_repaired.png	manage	47.7 K	2024-08-26 - 07:17	WillChatham
png	Unexpected_status_code_in_response_400_Bad_Request.png	manage	39.8 K	2024-02-13 - 14:50	WillChatham
png	Validation_results.png	manage	51.4 K	2023-11-27 - 12:03	WillChatham
png	Version_Resource_Usage.png	manage	11.6 K	2024-02-13 - 14:49	WillChatham
Copyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.