TRS Delta Monitoring Task (DMT) Troubleshooting

Authors: WilliamChatham
Build basis: V7.0.3 and V7.1.

The objective of page is to provide an overview of the TRS Delta Monitoring Task (DMT) and also include troubleshooting steps.

What is the TRS?

A Tracked Resource Set (TRS) exposes resources that allow clients to:

• Discover the exact set of resources.
• Track additions/removals.
• Monitor state changes.

Used in Engineering Lifecycle Management (ELM):

• Lifecycle Query Engine (LQE): Indexes data across projects for near-live reporting.
• Link Index Provider (LDX): Maintains links between artifacts in CM-enabled environments.

Event-Based vs Log-Based TRS

Type	Description	Availability	Benefits
Event-Based	Changes added to TRS changelog at time of change	Discontinued in V7.1	—
Log-Based	Changes recorded externally and processed into TRS	Available in V7.0.2+	Better consistency (Out of date configurations, skipped resources, missing/extra requirements and out of date requirements), error detection (Better Serviceability), and validation (Detect/correct a wider range of problems)

What is the Delta Monitoring Task (DMT)?

The DMT is a Java process within ERM for log-based TRS - the purpose is to query for and process three kinds of change (in this order) which update the TRS.

1. Archive/Restore compensations
2. Durable log changes
3. Container recovery

• The DMT gives Administrators a way to determine how up to date TRS processing is.
• DMT can be monitored by Administrators in the Tracked Resource Set monitoring page which shows the Watermark (How up to date the TRS processing is – this was never visible in event-based TRS
• The DMT runs every 2 seconds (non-configurable).
• It is normal for the DMT to be a few seconds behind and can go further behind when there is a lot of data to process
  • So this isn't necessarily a problem until it continues for a long time, perhaps >15 minutes

Monitoring TRS Currency

To access the Tracked Resource set go to :

 https://<JazzServerName>/rm/admin= &#8594; Debug &#8594; Tracked Resource Set  

Displays how far behind TRS processing is (days/hours/minutes/seconds).

DMT Debug Logging Setup

To set debug Logging:

1. Navigate to /server/conf/rm/log4j2.xml
2. Make a backup of this file before editing it.
3. Add the following to the Loggers section?

Edit log4j2.xml:

<!-- DMT Debug Logging -->
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsChangeProcessor" level="DEBUG"/>
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsDeltaMonitorTask" level="DEBUG"/>

<!--  Use debug to get CSV-formatted metrics from TRS -->
<Logger name="com.ibm.rdm.fronting.server.services.trs.ITrsService.metrics" additivity="false" level=“DEBUG">
<AppenderRef ref="trs_metrics"/></Logger>

DMT Debug Log Examples

• com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsDeltaMonitorTask

WARN: No watermark in the repository; compensations will remain pending until a watermark exists
DEBUG: DMT: begin/end compensating action
DEBUG: containerRecovery begin/end

• com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsChangeProcessor

DEBUG: DMT: Found %d version mapping changes in %s
DEBUG: Excluded container no longer being processed by AM.  Starting catch-up. %s
DEBUG: TrsChangeProcessor.commit: begin/end %d events from %s (normal)
DEBUG: TrsChangeProcessor.commit: %d compensating actions (compensation)

• com.ibm.rdm.fronting.server.services.trs.ITrsService.metrics

2025-03-11T04:26:40,767, watermark, 2025-03-11T11:26:32.911Z,1741692392911, 2025-03-11T11:26:35.097Z,1741692395097, 2025-03-11T11:56:32.911Z,1741694192911, 0,0, 2025-03-11T11:26:35.097Z,1741692395097, 0, 129, 5410
2025-03-11T04:26:42,939, watermark, 2025-03-11T11:26:35.097Z,1741692395097, 2025-03-11T11:26:42.814Z,1741692402814, 2025-03-11T11:56:35.097Z,1741694195097, 0,0, 2025-03-11T11:26:42.814Z,1741692402814, 0, 0, 94

DMT Performance Considerations

• Server load
• Large deliveries
• Database performance
• Archive/restore operations

Use IBM MustGather for diagnostics: IBM DOORS Next Generation MustGather

• Foundation Counters:

   https://<JazzServer>/rm/service/com.ibm.team.repository.service.internal.counters.ICounterContentService 

If there is a queue reported it can indicate load/wait times at the database level

• Resource Usage (Pools):

1. Check RM Statistics:
2. Gather Statistics:Navigate to https:///rm/debug
3. Click on Statistics

Identifying DMT Processing

• Archive/Restore: If the DMT is processing Archive/Restore requests the DMT page will display the ID of the configuration and the type of change (Archive/Restore)

• Large Change Sets:

Related topics: Deployment web home, Deployment web home

External links:

• IBM