TRS Validation

The objective of page is to provide an overview of the TRS Validation including troubleshooting.

Why Validate?

• A report is missing artifacts.
• A report includes outdated (stale) artifacts.
• A report shows the same artifact more than once, possibly with different attribute values or with its old and new name.
• An application view and a corresponding report in Jazz Reporting Service (JRS) show a different number of artifacts.
• A report includes deleted artifacts.
• LQE or LDX shows skipped resources.
  

See here for more details on validation:Validating TRS feeds and the LQE or LDX index

TRS Validation is also a much better alternative to performing a Full TRS rebase and LQE Datasource reindex which can take a long time to complete.

How to run an RM TRS Validation:

Running a TRS Validation may take a long time to complete and will put additional load on the server.

• We recommend running a validation when server usage is low, such as during a scheduled maintenance window.
• There is no way to stop a validation once it starts except by shutting down the RM server.
  • Note: The validation will not automatically resume when the server restarts, the validation must be started again after clearing the TRS cache (As its in an unknown state).
• Validation results are based on the data from when the validation starts, so if users make changes these changes will not be picked up and validated.
  • This is why we recommend running during a scheduled maintenance, otherwise, inaccurate validation results can be reported.
  • Running a validation during working hours is best to run against baselines only as these are immutable so cannot be affected by changes.

Note: If not already set, the DOORS Next TRS Validation requires the Selections Validation disk cache directory advanced property needs to be set before validation, pointing to a directory on the application server. To set this property, go to https://host:port/rm/admin and find the Advanced Properties page and allocate a directory with sufficient storage. We recommend that the Selections Validation disk cache directory is placed on a different file system than the DOORS Next server to prevent issues if the disk becomes full. In order to calculate the disk space requirements needed for the Selections Validation disk cache directory, you must use the Required file system space estimate tab on the RM TRS Feed Diagnostics page.

Determine the Selections Validation disk cache directory size steps:

1. Conduct a full rebase see Perform a Rebase and Reindex for IBM Engineering Requirements Management DOORS Next
2. When the rebase is complete, navigate to the TRS Feed Diagnostics page and select the Required file system space estimate tab to show the estimated directory size.
   

Due to the time involved running a TRS Rebase on production, we advise that you run the full RM TRS Rebase on a comparable test system. If a TRS Rebase is not possible, then you must monitor the disk space consumption during the validation operation and adjust storage allocation as necessary.

Procedure to run an RM TRS Validation:

Open the Engineering Requirements Management DOORS Next application administration page: https://host:port/rm/admin. On the toolbar, click TRS Feed Diagnostics. In the Validate dialog box, click Validate artifacts and configurations.

Validation options explained:

Validate configurations option:

When Validate all configurations option is selected, the following happens: TRS Validation performs a more thorough analysis of all configurations in the repository. Configuration membership is verified to agree with current state in the repository, and each configuration member is checked to ensure that it is present in the TRS. Note: This option must be used consistently whenever a validation is done. Otherwise, the validation cache may become stale and validation discrepancies will result. Those discrepancies can be resolved by using the Clear cached validation data option.

When Validate specific configurations option is selected, validation focuses only on the chosen configurations to validate, rather than all configurations and all resources in the TRS. It may be quicker to validate a few configurations than all configurations. Further, when you do not care about the validity of other configurations. You can select any combination of local and global configurations, and validation will check the union of all local configurations, which are contributed to each chosen global configuration.

Note: Notice that only configurations from the current application are checked. The membership of each local configuration is checked against the repository, and the TRS is checked to ensure that all configuration members are present.

Validate artifacts option:

Resolve problems option:

When there are unresolved discrepancies, you can see a list of

• Missing or extra artifacts.
• Unknown resource for Selections patch.
• Wrong patch eTag.
• Unexpected artifact added by a patch, etc.

The validation results are appended to the existing validation summary. You can view the number of repaired, missing, and extra artifacts in the feed. These artifacts are listed in separate sections.

1. If all issues are resolved, then the validation result shows a green check mark.
2. If there are discrepancies found that were fixed, then the validation result shows a Yellow Triangle.
3. If there are discrepancies to fix, then the validation result shows a red cross.

If you see discrepancies in reports after you validate the feed, consider running a full validation. This operation takes longer because it examines every entry in the TRS feed, discards all caches and starts with the beginning. The artifacts that cannot be repaired are listed as missing or extra artifacts. If you still see discrepancies, even after running the full validation, contact IBM Support.

Clear cached validation data option:

Tip: As you can validate either the TRS (with or without configurations) or specific configurations, it is important to use the same options each time. When validating specific configurations, options do not matter for the correctness of the next TRS validation.

Use this option in the following cases:

1. If the application server is restarted while a validation as running as it is in an unknown state.
2. After the TRS has been rebased or truncated, either when done manually, or as part of a scheduled periodic task.
3. Following a validation run with incompatible options, such as, after you accidentally cleared the Validate configurations checkbox but still want to keep validating configurations. If you suspect that previous validations reported incorrect failures due to inconsistent validation options, such as when you tried validating a configuration with an outdated configuration cache entry. (This might happen when you stop validating all configurations when you are validating TRS and then decide to validate specific.)
4. Selections Validation disk cache directory location change.
   
   

Throttling the TRS validation

TRS validation of the TRS providers, such as DOORS Next, or Engineering Test Management (ETM), is expected to be run as frequently as required. If the validation is regularly resolving problems when run monthly, or if you are experiencing user reported incidents that match the "Why Validate?" section above, then you need to validate more frequently, for example, weekly.

Whether to use throttling depends very much on your environment. Here are considerations, for evaluating whether you need to throttle:

• How many TRS providers do you need to validate within your IBM ELM ecosystem...at the same time?
• How many CPU do the machines running the TRS providers have?
• How many IBM HTTP Servers (IHS) do you have/How many IHS connections are available?
• What other off-hours automation are you running? This includes custom, in-house as well as say DCC jobs.
• Active Services is showing that each TRS validation thread is taking many minutes/hours for each thread

If, for example, you have multiple large ELM repositories of DOORS Next and ETM on the same shared resources, then you need to pay attention to the amount of time a validation takes/the overall CPU on the application server used and also if other users running their own scripts are complaining of longer durations to complete their scripts.

To throttle TRS validation feeds, you should:

• Login to https://host:port/{application}/admin
• Find Maximum threads for resource validation
• Find TRS Validation thread limit for repository queries
• Specify 25% of your available CPU for both these settings.

Note: the default for both values is 0, which equates to 100% of your CPU. The 25% recommendation is a starting point. Reduced CPU allocation could result in a longer time to complete. However, if the number of TRS validation threads was causing issues, then the overall time may well improve when throttled. Little's Law has been encountered when all maintenance jobs are scheduled off-peak for users, creating a new peak for maintenance.

When not to run validation

The Tracked Resource Set can become inconsistent in certain circumstances when configuration restores and archives are performed whilst the server is restarted. If there is a server restart during archiving or restoration of configurations, then you must run TRS Validate.

However, you should not validate during archive/restore, as you will be making changes that could cause the validate to report false positives, or fix (reverse) the action the restore/archive was performing. To know whether you can validate pr not, the administrator can check the current status via:

• The rm/admin -> Debug -> TRS Admin page where excluded containers will be listed in the watermark
• Check the rmTRS.log, to see if compensating actions are being reported in the log. You will need to set eTrsDeltaMonitorTask logger to DEBUG.
• Run select * form ARCHIVEMANAGER_AM_REQUEST, only validate if this returns 0 rows

Reading Validation results:

So in the example above:

• Out of 4788 Artifacts validated 4786 were successful and 2 failed validation and those 2 were resolved.
• Out of 27 Selections validated 13 were successful and 14 failed validation and could not be resolved so will need to be reviewed.
  
  

Troubleshooting:

Logging:

The TRS Validation logs out to the rm.log which can roll over very quickly on a busy server so its best to have the validation log out to its own log (TRS-Validation.log). You can do this by adding the following to the log4j2.xml located here: [installDir]/server/conf/rm/. Log4j2.xml changes are dynamically read, so there is no need to restart the server.

In the Appenders section, define a new appender by introducing:

<RollingFile name="rmTRSValidation" filename="${dir}/${app}TRS-Validation.log" filePattern="${dir}/${app}TRS-Validation-%i.log">
   <PatternLayout pattern="${pattern}"/>
   <Policies>
      <SizeBasedTriggeringPolicy size="10 MB"/>
   </Policies>
   <DefaultRolloverStrategy max="5"/>
</RollingFile>

In the Loggers section, include the following three loggers:

<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.validation" additivity="false" level="INFO">
   <AppenderRef ref="rmTRSValidation"/>
</Logger>
<Logger name="com.ibm.team.repository.service.internal.ReportsDiagnosticsRestService" additivity="false" level="INFO">
   <AppenderRef ref="rmTRSValidation"/>
</Logger>
<Logger name="com.ibm.team.repository.service.internal.trs" additivity="false" level="INFO">
   <AppenderRef ref="rmTRSValidation"/>
</Logger>

TRS Validation Results Downloader:

Steps to run the TRS Validation Results Downloader tool below:

1. If not already installed follow the steps in the provided read me file to install the tool.
2. To run the tool navigate to rm/admin and open the Debug tab followed by selecting L3 Tools.
3. Select TRS Validation Results Downloader
4. First input nothing into the fields, just press Download
5. This will provide the list of previous validations and includes a UUID Token for each validation.
6. Copy the UUID Token for the TRS Validation you wish to check, e.g. "_5VO0kACVEe-xSu7KysdNHg"
7. Paste this UUID Token into the Token field or leave blank if you wish to download all the available validations results.
8. For the location select a folder on the RM server where the results can be written to e.g. C:\IBM\JazzTeamServer_7.0.2SR1\server\Validation
9. Click on Download
10. The results will be written to the C:\IBM\JazzTeamServer_7.0.2SR1\server\Validation folder and a separate folder is created for each validation denoted by the Token UUID.

The results break down as follows:

• summary.txt : A Summary of the Validation results
• extra.txt : Lists any extra resources reported in the validation.
• missing.txt : Lists any missing resources reported in the validation.
• failures.txt : Lists any failures reported in the validation.
• repair.txt : Lists any repairs made during the validation.
• selectionMissing.txt : Lists any missing selection resources reported in the validation.
• selectionExtra.txt : Lists any extra selection resources reported in the validation.
• selectionRepair.txt : Lists any Selection repairs made during the validation.

Known validation errors:

Selections URLs repeating in successive validations

Possible causes

• Pre-iFix025: PH56481 Defect 145950 DOORS Next TRS validation cannot repair selections when TRS is running in event-based mode means repair of selections doesn't always work, so a selection which should have been repaired in one validation is also present in subsequent validations. this issue was fixed in V7.0.2 ifix025 - see the workaround below

• Post-iFix025: If you hit this issue create an IBM support case.

Workaround:

• Pre-iFix025, Contact IBM Support and request the L3 Selections Tool (v1.0.9 or greater) and Copy/Paste the repeated selection URL(s) from the validation report into the L3 TRS Selections Validate/Repair tool, using Reset+Repair.

1. Follow the install instructions in the read me.
2. From the /rm/admin -> Debug page, select "L3 Services" and select TRS Selections Validation/Repair tool from the drop down list.
3. Specify the URLs of the configurations to be validated or repaired in the text area. Copy and paste the ~Selections URL(s) on its own line. The URLs can be encoded or decoded.
4. Select "Reset (non-recursive; recreates the selection resources for the configurations in scope regardless of the selection status)" and "Repair"

Unexpected status code in response: 400: Bad Request:

Possible causes

• Possible problematic Artifact.
• If the event states its “UnBlocked” it was removed from the ignore list which tries to process skipped resources again.

Workaround:

• Possibly previously ignored Skipped resources unignored. Confirm if this was done.
• New capability in the /admin  Debug  Track Resource Set we can use to check a Versions resource to see which configuration its part of and if it’s not part of a config it can be ignored.

Selections page ... cannot be retrieved. (Unexpected status code in response: 401: Unauthorized):

Possible causes:

• Possible transient issue (temporary issue with the proxy Retry tends to work similar to 401s seen in LQE Skipped resources.
  • To retry do a specific validation on the affected Stream/Baseline.
  • Make sure The Cache is cleared.
• Possible issue with the URL (Percentage signs in the URL) Not a validation issue instead a Proxy issue, see if the below setting is set. See: Configuring IBM HTTP Server as a reverse proxy for WebSphere Application Server.
  • Optional: Additional configuration for application ETL data collection jobs
  • If your application reporting ETL jobs are configured to use Jazz® Team Server (oAuth) authentication and there is an HTTP fronting server in the application topology, you must add an additional entry to the httpd.conf file.
  • Navigate to the installation directory for your IBM HTTP Server.
  • Open conf\httpd.conf in an editor.
  • In the Global section (Section 1) of the file, add this entry:
    • SetEnv websphere-nocanon 1
    • Restart your web server. |

• Retry the validation for the specific configuration (make sure to clear the cache).

Too many Selections discrepancies. Validation process for the Selections resource interrupted

Workaround: Contact IBM Support and request the L3 Selections Tool (v1.0.9 or greater) and perform a reset/repair for the affected selection

To run the L3 Selections Tool (Provided by IBM Support) to workaround the issue do the following:

1. Follow the install instructions in the read me.
2. From the /rm/admin -> Debug page, select "L3 Services" and select TRS Selections Validation/Repair tool from the drop down list.
3. Specify the URLs of the configurations to be validated or repaired in the text area. Copy and paste the ~Selections URL(s) on its own line. The URLs can be encoded or decoded.
4. Select "Reset (non-recursive; recreates the selection resources for the configurations in scope regardless of the selection status)" and "Repair"

Missing and Extra resources reported in validation results (False positives):

Root Cause: Issue will be fixed under: Known Issue: DT363651: ERM TRS Validation reports false positive Extra and missing resources in the feed (WI 146949) Scheduled to be fixed in V7.0.2 ifix031 and V7.0.3 ifix008

Workaround: To detect if you are seeing false positives contact IBM Support and request the TRS Validation Results Downloader tool (see here for more details:TRS Validation Results Downloader ). This tool allows the Administrator to download the results of the validation and has the added benefit of being able to report if there are false positives.

If the tool reports missing.txt or extra.txt along with the Summary.txt then these Extras and missing are false positives and can be ignored.) So if there is no repaired.txt file in the results then they are all false positives.

TRS Validation results Summary reports Selection failures that are repaired:

This was raised under DT260140: TRS Validation results Summary reports Selection failures that are repaired. (WI 572188). The TRS Validation results Summary is incorrect and reports issues with “Selections failed validation” when they were actually repaired when reviewed in the full validation results.

Scheduled to be fixed in Log based TRS Only in V7.0.2 ifix031 and V7.0.3 ifix009

Extras in selections in validation results:

Variant 2 Log based/Event based: GUI shows: ID CRRRW7552E The specified artifact cannot be found in the current configuration. Unable to load

Root cause: Project area with no active components (All archived) returns an error in TRS Validation, GUI shows: ID CRRRW7552E The specified artifact cannot be found in the current configuration. Unable to load. Technically, you can archive all configurations in a component as there is nothing to prevent a User from doing that. However there is no value in having such a component in a system. Issue will be fixed under: Known Issue: DT269559: TRS Validation reports a component as Extra in the feed due to no active streams existing in a component (WI 146822) Scheduled to be fixed in V7.1.



Workaround: The extra can either be ignored or else Unarchive the component and rerun the TRS Validation.

Missing from Selections in validation results:

Event with patch for selections resource unknown to validation in validation results:

Root cause unknown.

Workaround 1: Perform a validation on the affected configuration, see steps below:

1. For each URL you will need to identify the Stream name so in the example above edit the URL: https://XXXXX/rm/configSelections/stream/_Xz9RIIRxEe6Gzpvqit1zkg as https://XXXXX/rm/cm/stream/_Xz9RIIRxEe6Gzpvqit1zkg and open the URL and take note of the stream name.
2. In the TRS Validation page select "Validate specific configurations" and select the affected configurations.
3. Check the "Clear cached validation data" option.
4. Run the Validation.

NOTE: If you "Clear cached validation data" within the context of a specific configuration, then you only clear the cache data of that single configuration. This will not clear the entire cache.

Workaround 2: An alternative, for non log-based RM systems is to use the TRS Selections Validation/Repair tool to process multiple URIs concurrently.

To run the L3 Selections Tool (Provided by IBM Support) to workaround the issue do the following:

1. Follow the install instructions in the read me.
2. From the /rm/admin -> Debug page, select "L3 Services" and select TRS Selections Validation/Repair tool from the drop down list.
3. Specify the URLs of the configurations to be validated or repaired in the text area. Copy and paste the ~Selections URL(s) on its own line. The URLs can be encoded or decoded.
4. Select "Reset (non-recursive; recreates the selection resources for the configurations in scope regardless of the selection status)" and "Repair"

Addition of unexpected artifact by Selections patch:

Root cause: This is an issue where the problem was identified and fixed but is still reported as an issue in the Validation results, see this in the example screen print below:



This will be covered in defect, PH59596: (WI572188 ) TRS Validation results Summary reports Selection failures that are repaired. Scheduled to be fixed in V7.0.2 ifix031 and V7.0.3 ifix009.

Workaround: Can be ignored if the Stream appears as repaired in the results.

Missing Deletion by selections patch in validation results:

Root cause unknown.

Note We have seen this appear under the validation results but its benign and does not affect Customer reporting so can be ignored.

This stopwatch is already stopped reported in the rmTRS.log

If this message is encountered in the logs, contact IBM Support. This is known issue DT387232 - IllegalstateException: This stopwatch is already stopped in log-based mode. Fixed in V7.0.2 ifix030 and V7.0.3 ifix007.

 2024-07-03T18:28:17,259+0200 [TID: 78881AB9][lqe_user][Default Executor-thread-1660135 @@ 18:28 <unauthenticated> <LQE/1.0@1.1.1.2> /rm/configSelections/baseline/_-zbafdjhkdszO9PGMcasjA/query] ERROR com.ibm.rdm.fronting.server.services.trs           - Error processing HTTP request

   com.ibm.rdm.fronting.server.services.trs.ConfigSelectionsRestService@93035083

   This stopwatch is already stopped.
...
java.lang.IllegalStateException: This stopwatch is already stopped.

If encountered pre V7.0.2 ifix030, then the server will need to be restarted and any selection/baseline republished.

CRJAZ3160E The validation was canceled since XXX problems were encountered. Only 100000 can be processed at once:

2023-11-27T15:59:32,056+0200 [TID: XXXXX][][ TRS validation thread 0]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task
com.ibm.team.repository.common.TeamRepositoryException: CRJAZ3160E The validation was canceled since 131172 problems were encountered. Only 100000 can be processed at once. The TRS feed may be damaged and should be rebuilt in its owning application.
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils$ValidationProcessor.validateTrsUris(InternalTrsValidationUtils.java:510) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils.validateTrsUris(InternalTrsValidationUtils.java:368) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService.lambda$0(InternalTrsValidationUtilsService.java:269) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidation.validate(InternalTrsValidation.java:64) ~[repository_service.jar:?]
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]
   at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]
   at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
   at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
   at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
   at com.sun.proxy.$Proxy144.validate(Unknown Source) ~[?:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService.lambda$1(InternalTrsValidationUtilsService.java:285) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService$4.run(InternalTrsValidationUtilsService.java:571) [bundleFile:?]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) [?:1.8.0]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:1.8.0]
   at java.lang.Thread.run(Thread.java:825) [?:2.9 (04-17-2023)]

The 100,000 limit can be increased. by following the procedure below:

1. Under the rm/admin advanced properties set Maximum TRS errors to display to greater than the amount detailed in the error, if the number of errors are above 500,000 please contact IBM support for assistance.
2. Under the rm/admin advanced properties set TRS validation repair phase checks for false positives to false (if there are 2 properties with the same name, set them both to false).
3. Rerun the validation.
4. When it completes navigate to the rm/admin advanced properties and reset TRS validation repair phase checks for false positives back to true.
   
   

Failed deleting TRS Selection cache file:

This issue can occur if there are issues with the Selections Validation disk cache directory, if any files were manually deleted or manipulated the Validation will fail and the following error is logged.

2023-12-11T10:25:19,149+0100 [TID: XXXXXX][][TRS validation thread 0]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task
com.ibm.team.repository.common.TeamRepositoryException: Failed deleting TRS Selection cache file /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache/selections_Selections.baseline.XXXXXXXXXXXXXXXXXX
   at com.ibm.team.repository.service.internal.trs.validation.selCache.TrsSelectionsCacheService.clearCachedSelections(TrsSelectionsCacheService.java:266) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.selCache.TrsSelectionsCacheService.clearAllCachedSelections(TrsSelectionsCacheService.java:284) ~[bundleFile:?]
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]
   at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]
   at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
   at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
   at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
   at com.sun.proxy.$Proxy145.clearAllCachedSelections(Unknown Source) ~[?:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils$ValidationProcessor.dropAllCaches(InternalTrsValidationUtils.java:584) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils$ValidationProcessorForStrings.loadCacheOrReadActiveMembers(InternalTrsValidationUtils.java:799) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils$ValidationProcessor.validateTrsUris(InternalTrsValidationUtils.java:441) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtils.validateTrsUris(InternalTrsValidationUtils.java:368) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService.lambda$0(InternalTrsValidationUtilsService.java:269) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidation.validate(InternalTrsValidation.java:64) ~[repository_service.jar:?]
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]
   at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]
   at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
   at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
   at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56) ~[org.eclipse.soda.sat.core_1.1.0.201611021629.jar:?]
   at com.sun.proxy.$Proxy144.validate(Unknown Source) ~[?:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService.lambda$1(InternalTrsValidationUtilsService.java:285) ~[bundleFile:?]
   at com.ibm.team.repository.service.internal.trs.validation.InternalTrsValidationUtilsService$4.run(InternalTrsValidationUtilsService.java:571) [bundleFile:?]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) [?:1.8.0]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:1.8.0]
   at java.lang.Thread.run(Thread.java:825) [?:2.9 (04-17-2023)]
Caused by: java.nio.file.NoSuchFileException: /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache/selections_Selections.baseline.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
   at sun.nio.fs.UnixException.translateToIOException(UnixException.java:98) ~[?:1.8.0]
   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:114) ~[?:1.8.0]
   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:119) ~[?:1.8.0]
   at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:256) ~[?:1.8.0]
   at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:115) ~[?:1.8.0]
   at java.nio.file.Files.delete(Files.java:1137) ~[?:1.8.0]
   at com.ibm.team.repository.service.internal.trs.validation.selCache.TrsSelectionsCacheService.clearCachedSelections(TrsSelectionsCacheService.java:263) ~[bundleFile:?]
   ... 28 more

We cache in memory the index file of the Selections cache directory. If the cache files are manually deleted while the server was running, then this issue can occur and can only be resolved with a DOORS Next server restart.

Selections patch events in Change Log do not have expected beforeETag 1.ab1cd234.

The Validation result will show a Selection resource as flagged, when running against a specific configuration.

The result will state:

https://servername/rm/configSelections/stream/_1a2BcD3eFghiLJ3XCVU6kA   Selections patch events in Change Log do not have expected beforeETag 1.ab1cd234.

This may happen for two reasons:

1. TRS validation runs were inconsistent. See: Validating TRS feeds and the LQE or LDX index
2. TRS feed has a broken sequence of TRS patches for this particular Selections.

The recommendation in this case to validate this particular configuration using the Clear Cache option.

Invalid TRS response.. Unexpected status code from GET.. 502: Bad Gateway

This Validation failure due to 502 Bad Gateway error can occur due to server load with connections.

2023-11-24T15:01:03,804+0100 [TID: EB4D1F93][][       TRS validation thread 3]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task com.ibm.team.repository.service.internal.trs.feed.TrsFeedReadException: Invalid TRS response at <https://xxxx/rm/trs2ChangeLogs/_HIZhAXl0Ee6n7Ow9oWyCxw>;
...
Caused by: com.ibm.team.repository.service.internal.trs.feed.impl.TrsResponseException: Unexpected status code from GET to <https://xxxx/rm/trs2ChangeLogs/_HIZhAXl0Ee6n7Ow9oWyCxw>; 502: Bad Gateway
   at com.ibm.team.repository.service.internal.trs.feed.impl.Trs20FeedReaderService$TrsRdfResponse.create(Trs20FeedReaderService.java:676) ~[bundleFile:?]

We recommend running the validation during a quiet period with less load on the servers and connections. So you can attempt a rerun during a quiet period.

Selections page https://rm/configSelections/stream/xxx/journal/xxx> Cannot be retrieved. (Unexpected status code in response: 410:Gone)

Validation failure can occur if the a stream selection has a problem retrieving a journal page, the selection is flagged as invalid by the validation but the validation cannot repair this.

This identified as a defect and raised under PH58915: OOTB TRS VALIDATION CANNOT REPAIR SELECTIONS WITH MISSING JOURNAL PAGES (WI 146583) . Fixed in V7.0.2 ifix028 and V7.0.3 ifix004.

Workaround: Contact IBM Support and request the L3 Selections Tool (v1.0.9 or greater) and perform a reset/repair for the affected selection (Note: this also requires defect PH58915: OOTB TRS VALIDATION CANNOT REPAIR SELECTIONS WITH MISSING JOURNAL PAGES (WI 146583) to be applied. Issue is fixed in V7.0.2 ifix027).

To run the L3 Selections Tool (Provided by IBM Support) to workaround the issue do the following:

1. Follow the install instructions in the read me.
2. From the /rm/admin -> Debug page, select "L3 Services" and select TRS Selections Validation/Repair tool from the drop down list.
3. Specify the URLs of the configurations to be validated or repaired in the text area. Copy and paste the ~Selections URL(s) on its own line. The URLs can be encoded or decoded.
4. Select "Reset (non-recursive; recreates the selection resources for the configurations in scope regardless of the selection status)" and "Repair"

No space left on device

com.ibm.team.repository.service.internal.trs.validation.resource.ResourceValidationException: Validating TRS change log in 'com.ibm.rdm.fronting.server.trs.validation': java.io.IOException: No space left on device

One problem with this message, is the subsequent messages that you get when trying to run TRS Validate. This occurs even if Clear Configuration Cache is selected:

2023-12-30T05:41:32,854+0100 [TID: 1544B173][][       TRS validation thread 2]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task

com.ibm.team.repository.service.internal.trs.validation.resource.ResourceValidationException: Validating TRS change log in 'com.ibm.rdm.fronting.server.trs.validation': Failed deleting TRS Selection cache file /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache/selections_configSelections.baseline._k3sOYQp5Ee6yhpy2leP1DA._393oe6KwEe6pO7zRQeW-4g
   at com.ibm.team.repository.service.internal.trs.validation.resource.ResourceValidationExceptionCatcher.close(ResourceValidationExceptionCatcher.java:120) ~[bundleFile:?]

Root cause: Issue occurs due to a known issue: DT379224:TRS Validation fails due to Error encountered Validating TRS change log in 'com.ibm.rdm.fronting.server.trs.validation': Failed deleting TRS Selection cache file /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache/selections_configSelections.xxxxxxx(574208). This often occurs if Rebase<\i> has not been performed, or was performed some time ago. The DOORS Next TRS Feed Diagnostics page also shows Required file system space estimate tab on the TRS Feed Diagnostics page for DOORS Next resources that should be adhered to, to avoid this disk full scenario.

Workaround for insufficient Validation Cache size: Manually clear the cache (Delete the contents of the Selections Validation disk cache directory advanced property Location e.g. /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache folder) and restart the RM server. The subsequent attempt to run TRS validate will succeed.
If your investigation of disk space shows that there is sufficient space in the Validation Cache directory, then continue to evaluate the /tmp directory, or the temp space. If this is deemed too small, then change the location that DOORS Next will use by applying the following workaround.

Workaround for insufficient temp space

• cd [ELM_Install]/server
• edit server.startup script.
• Change the following line to add your new path:
  • set JAVA_OPTS=%JAVA_OPTS% -Dcom.ibm.team.repository.tempDir="%TEMP%"

2024-02-20T02:02:117,117+0100 [TID: XXXXXXXX][][ TRS validation thread 1] WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task
com.ibm.team.repository.service.internal.trs.feed.TrsFeedReadException: Cutoff event(s) <[urn:_TOgnsOiVEeyOKKA0JZrFGA:1700273627117]> not found in Change Log

2024-02-23T14:02:52,873+0000 [TID: 3442C500][][ TRS validation thread 2] ERROR ervice.internal.trs.validation.ContentFormatHelper - Error encountered during validation
com.ibm.team.repository.service.internal.trs.feed.TrsFeedReadException: Cutoff event(s) <[urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:normal:E]> not found in Change Log. 

Starting execution at Wed Feb 28 14:19:22 UTC 2024

search(urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E)
TRS Types Helper v1.0.0
searchForTypeInfo(urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E)
Searching urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E ...
Wed Feb 28 14:19:27 UTC 2024 -> 100 changelog pages processed so far
Wed Feb 28 14:19:32 UTC 2024 -> 200 changelog pages processed so far
Wed Feb 28 14:19:37 UTC 2024 -> 300 changelog pages processed so far
Wed Feb 28 14:19:41 UTC 2024 -> 400 changelog pages processed so far
Wed Feb 28 14:19:47 UTC 2024 -> 500 changelog pages processed so far
Wed Feb 28 14:19:52 UTC 2024 -> 600 changelog pages processed so far
Wed Feb 28 14:19:56 UTC 2024 -> 700 changelog pages processed so far
Wed Feb 28 14:20:02 UTC 2024 -> 800 changelog pages processed so far
Wed Feb 28 14:20:06 UTC 2024 -> 900 changelog pages processed so far
Wed Feb 28 14:20:12 UTC 2024 -> 1000 changelog pages processed so far

METRICS

searchForEvent [
Total calls: 1
Avg: 64070
Min: 64070
Max: 64070]
RESULTS FOR urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E
NOT found, urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E
Wed Feb 28 14:20:27 UTC 2024, 1303 changelog pages processed
Date of current changelog page, Wed Feb 28 14:17:13 UTC 2024
Date of last processed changelog page, Tue Feb 13 16:23:42 UTC 2024
Done searching for information on urn:publicid:8d660674-ddbd-47ec-bbf4-ff0a5c4562f3:NORMAL:E

Failed to get the result of this valdation
A server error occurred: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body>
 <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> 
<p>Please contact the server administrator at you@your.address to inform them of the time this error occurred, and the actions you performed just before this error.</p> 
<p>More information about this error may be available in the server error log.</p> <hr> 
<address>IBM_HTTP_Server Server at servername.ibm.com Port 443</address> </body></html> - Error code 500

The validation was canceled because of an error. For details, check the server log file.

2024-01-28T21:48:20,567+0100 [TID: F6BBD78F][][       TRS validation thread 0]  INFO e.internal.trs.validation.TrsCachePopulationManger - Reading fetched TRS data for https://xxx/rm/trs2

2024-01-29T08:00:07,289+0100 [][TRS-Resource-Validation: validate-trs-resource <https://xxx/rm/configSelections/baseline/_K2BToH3lEe6CGdsfON46Tg>;]  INFO .validation.selCache.RegularFeedPropertiesSupplier - Selections Cache Storage Location for feed id "com.ibm.rdm.fronting.server.trs.validation" is /opt/IBM/JazzTeamServer/server/conf/rm/ValidationCache

2024-01-31T04:38:19,948+0100 [TID: F6BBD78F][][       TRS validation thread 0]  WARN l.trs.validation.InternalTrsValidationUtilsService - Error running task

com.ibm.team.repository.service.internal.trs.validation.resource.ResourceValidationException: Repairing Selections resources in 'com.ibm.rdm.fronting.server.trs.validation': java.util.ConcurrentModificationException: java.util.ConcurrentModificationException: java.util.ConcurrentModificationException