TRS Delta Monitoring Task (DMT) Troubleshooting

Authors: WilliamChatham
Build basis: V7.0.3 and V7.1.

The objective of page is to provide an overview of the TRS Delta Monitoring Task (DMT) and also include troubleshooting steps.

What is the TRS?

A Tracked Resource Set (TRS) exposes resources that allow clients to:

• Discover the exact set of resources.
• Track additions/removals.
• Monitor state changes.

Used in Engineering Lifecycle Management (ELM):

• Lifecycle Query Engine (LQE): Indexes data across projects for near-live reporting.
• Link Index Provider (LDX): Maintains links between artifacts in CM-enabled environments.

Event-Based vs Log-Based TRS

Type	Description	Availability	Benefits
Log-Based	Changes recorded externally and processed into TRS	Available in V7.0.2+	Better consistency (Out of date configurations, skipped resources, missing/extra requirements and out of date requirements), error detection (Better Serviceability), and validation (Detect/correct a wider range of problems)
Event-Based	Changes added to TRS changelog at time of change	Discontinued in V7.1	—

What is the Delta Monitoring Task (DMT)?

The DMT is a Java process within ERM for log-based TRS - the purpose is to query for and process three kinds of change (in this order) which update the TRS.

1. Archive/Restore compensations
2. Durable log changes
3. Container recovery

• The DMT gives Administrators a way to determine how up to date TRS processing is.
• DMT can be monitored by Administrators in the Tracked Resource Set monitoring page which shows the Watermark (How up to date the TRS processing is – this was never visible in event-based TRS
• The DMT runs every 2 seconds (non-configurable).
• It is normal for the DMT to be a few seconds behind and can go further behind when there is a lot of data to process
  • So this isn't necessarily a problem until it continues for a long time, perhaps >15 minutes

Monitoring TRS Currency

To access the Tracked Resource set go to :

 https://<JazzServerName>/rm/admin/#action=com.ibm.rdm.fronting.server.web.trs

Displays how far behind TRS processing is (days/hours/minutes/seconds).

DMT Debug Logging Setup

To set debug Logging:

1. Navigate to /server/conf/rm/log4j2.xml
2. Make a backup of this file before editing it.
3. Add the following to the Loggers section?

Edit log4j2.xml:

<!-- DMT Debug Logging -->
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsDeltaMonitorTask" level="DEBUG"/>
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsChangeProcessor" level="DEBUG"/>
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.logbased.ContainerChangeProcessor" level="DEBUG"/>
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.feed.TrsChangeLogService" level="DEBUG"/>


<!--  As this is extremely verbose, only set this to Debug when/if waiting for pending change is present in the logs for a "long time", gather an hours log entries at DEBUG level and then set back to INFO.-->
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.logbased.InflightUpdates" level="DEBUG"/>


<!--  Use debug to get CSV-formatted metrics from TRS -->
<Logger name="com.ibm.rdm.fronting.server.services.trs.ITrsService.metrics" additivity="false" level="DEBUG">
<AppenderRef ref="trs_metrics"/></Logger>

DMT Debug Log Examples

• com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsDeltaMonitorTask

WARN: No watermark in the repository; compensations will remain pending until a watermark exists
DEBUG: DMT: begin/end compensating action
DEBUG: containerRecovery begin/end

• com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsChangeProcessor

DEBUG: DMT: Found %d version mapping changes in %s
DEBUG: Excluded container no longer being processed by AM.  Starting catch-up. %s
DEBUG: TrsChangeProcessor.commit: begin/end %d events from %s (normal)
DEBUG: TrsChangeProcessor.commit: %d compensating actions (compensation)

• com.ibm.rdm.fronting.server.services.trs.internal.logbased.ContainerChangeProcessor

2025-03-11T20:16:33,162+0200 [][TRS DMT] DEBUG ces.trs.internal.logbased.ContainerChangeProcessor - Processing configuration change CREATE on com.ibm.team.repository.service.internal.vvc.vvcmodel.impl.ConfigurationImpl@531d9de7 (stateId: [UUID _WlH7wDc2EfGOVqXvcnmz7w], itemId: [UUID _Vzm_QDc2EfGOVqXvcnmz7w], origin: <unset>, immutable: true) (contextId: [UUID _8lNyYNwSEd2pIJ5QVwgQGg], modified: 2025-03-10 14:43:20.828, workingCopy: <unset>) (predecessor: <unset>) (id: https://jazz.ibm.com/rm/cm/stream/_Wcm_QDc2EfGOVqXvcnmz7w, name: TEST_Configuration_XXX, description: null, creationTime: 2025-03-10 14:43:15.64, type: workspace, archived: false, context: null)
2025-03-11T20:16:33,162+0200 [][TRS DMT] DEBUG ces.trs.internal.logbased.ContainerChangeProcessor - Container ContainerChange [timestamp=2025-03-10 14:43:15.64, type=CREATE, item=com.ibm.team.repository.service.internal.vvc.vvcmodel.impl.ConfigurationImpl@531d9de7 (stateId: [UUID _XlH7wDc2EfGOVqXvcnmz7w], itemId: [UUID _Vzm_QDc2EfGOVqXvcnmz7w], origin: <unset>, immutable: true) (contextId: [UUID _8lNyYNwSEd2pIJ5QVwgQGg], modified: 2025-03-10 14:43:20.828, workingCopy: <unset>) (predecessor: <unset>) (id: https://jazz.ibm.com/rm/cm/stream/_Wcm_QDc2EfGOVqXvcnmz7w, name: TEST_Configuration_XXX, description: null, creationTime: 2025-03-10 14:43:15.64, type: workspace, archived: false, context: null)] CREATE request at:2025-03-10 14:43:15.64
2025-03-11T20:16:33,162+0200 [][TRS DMT] DEBUG ces.trs.internal.logbased.ContainerChangeProcessor - Configuration (stream=true) Wcm_QDc2EfGOVqXvcnmz7w create at: 2025-03-10 14:43:15.64
2025-03-11T20:16:33,162+0200 [][TRS DMT] DEBUG ces.trs.internal.logbased.ContainerChangeProcessor - Processed configuration change CREATE on com.ibm.team.repository.service.internal.vvc.vvcmodel.impl.ConfigurationImpl@531d9de7 (stateId: [UUID _WlH7wDc2EfGOVqXvcnmz7w], itemId: [UUID _Vzm_QDc2EfGOVqXvcnmz7w], origin: <unset>, immutable: true) (contextId: [UUID _8lNyYNwSEd2pIJ5QVwgQGg], modified: 2026-04-13 14:43:20.828, workingCopy: <unset>) (predecessor: <unset>) (id: https://jazz.ibm.com/rm/cm/stream/_Vzm_QDc2EfGOVqXvcnmz7w, name: EBS_L2_SYSP_Configuration_Project_OVERLANDAI_ORV_MK_120_DG-101103, description: null, creationTime: 2025-03-10 14:43:15.64, type: workspace, archived: false, context: null)
2025-03-11T20:16:33,162+0200 [][TRS DMT] DEBUG .services.trs.internal.logbased.TrsChangeProcessor - Item [_-4rQUGh7EfCrGMGVoeJ6BA._-5wOYWh7EfCrGMGVoeJ6BA] not analyzed because timestamp matches configuration timestamp : https://jazz.ibm.com/rm/cm/stream/_Wcm_QDc2EfGOVqXvcnmz7w.

• com.ibm.rdm.fronting.server.services.trs.internal.feed.TrsChangeLogService (This is extremely verbose as it logs each transaction the DMT is processing, "CREATION" ,"DELETION", "MODIFICATION","ARCHIVE" and "RESTORE".

2025-03-11T20:16:47,417+0200 [][RM-TRS Event Processing (serial)-1] DEBUG ver.services.trs.internal.feed.TrsChangeLogService - TRS 2.0 addChangeLogEvents( eventType=CREATION, url=https://jazz.ibm.com/rm/versionedResources/BI_JvLr0PzlEfClPdo9upNKPQ/state/_zY4msQJfEfGLk4Rz5jos_w/core/TX_JwprdfzlEfClPdo9upNKPQ/state/_AAqlcDc4EfGOVqXvcnmz7w, rationale = NORMAL, etag = null )
2025-03-11T20:16:47,417+0200 [][RM-TRS Event Processing (serial)-1] DEBUG ver.services.trs.internal.feed.TrsChangeLogService - TRS 2.0 addChangeLogEvents( eventType=MODIFICATION, url=https://jazz.ibm.com/rm/configSelections/stream/_s0vAltbxEeqkc-58C6IiSQ, rationale = NORMAL, etag = 1.4a5d757f, patch= { beforeETag=1.cb681fa9, beforeHashCode=-1734854246194175041, afterETag=1.4a5d757f, afterHashCode=-6117785062366859013, rdfPatch='

• com.ibm.rdm.fronting.server.services.trs.ITrsService.metrics

2025-03-11T04:26:40,767, watermark, 2025-03-11T11:26:32.911Z,1741692392911, 2025-03-11T11:26:35.097Z,1741692395097, 2025-03-11T11:56:32.911Z,1741694192911, 0,0, 2025-03-11T11:26:35.097Z,1741692395097, 0, 129, 5410
2025-03-11T04:26:42,939, watermark, 2025-03-11T11:26:35.097Z,1741692395097, 2025-03-11T11:26:42.814Z,1741692402814, 2025-03-11T11:56:35.097Z,1741694195097, 0,0, 2025-03-11T11:26:42.814Z,1741692402814, 0, 0, 94

DMT Performance Considerations

• Server load
• Large deliveries
• Database performance
• Archive/restore operations

Use IBM MustGather for diagnostics: IBM DOORS Next Generation MustGather

• Foundation Counters:

   https://<JazzServer>/rm/service/com.ibm.team.repository.service.internal.counters.ICounterContentService 

If there is a queue reported it can indicate load/wait times at the database level

• Resource Usage (Pools):

1. Check RM Statistics:
2. Gather Statistics:Navigate to https:///rm/debug
3. Click on Statistics

Identifying DMT Processing

• Archive/Restore: If the DMT is processing Archive/Restore requests the DMT page will display the ID of the configuration and the type of change (Archive/Restore)

• Large Change Sets:

DMT Known issues:

List of Known issues that are known to affect DMT:

Known issue	Description	Fixed in/Scheduled for
DT469185	Unknown container replacement can result in blocking the TRS from updating resulting in TRS lag (151520)	Scheduled for V7.1 ifix011, v7.2 ifix003 and V7.0.3 ifix023
DT468525	TRS DMT: registrations must be removed when the transaction completes (598358) See Flash on this DOORS Next Log Delta Monitoring Task (DMT) may stall resulting in processing delays (Affects Log based only)	Fixed in V7.1 ifix010 and scheduled for v7.2 ifix004 and V7.0.3 ifix023
DT468092	TRS DMT does not process compensation events promptly (151504)	Scheduled for V7.1 ifix011, v7.2 ifix004 and V7.0.3 ifix023
DT453905	Tracked Resource Set(TRS) Delta Monitor task(DMT) lag resulting in poor query performance in DMT (150392)	V7.1 ifix009 (Scheduled for V7.0.3 ifix023 and V7.2 ifix003)
DT454904	IChangeSetService.getEarliestPendingChangeSetCommitTime() returns out-of-date timestamps (591959)	V7.0.3 ifix022, V7.1 ifix007, V7.2 GA
DT457527	TRS Archive Error : Unknown container replacement can result in blocking the TRS from updating resulting in stale data (150715)	V7.0.3 ifix022, V7.1 ifix009, V7.2 ifix002

Related topics: Deployment web home, Deployment web home

External links:

• IBM