TRS Delta Monitoring Task (DMT) Troubleshooting

Authors: WilliamChatham
Build basis: V7.0.3 and V7.1.

The objective of page is to provide an overview of the TRS Delta Monitoring Task (DMT) and also include troubleshooting steps.

What is the TRS?

A Tracked Resource Set (TRS) exposes resources that allow clients to:

• Discover the exact set of resources.
• Track additions/removals.
• Monitor state changes.

Used in Engineering Lifecycle Management (ELM):

• Lifecycle Query Engine (LQE): Indexes data across projects for near-live reporting.
• Link Index Provider (LDX): Maintains links between artifacts in CM-enabled environments.

Event-Based vs Log-Based TRS

Type	Description	Availability	Benefits
Event-Based	Changes added to TRS changelog at time of change	Discontinued in V7.1	—
Log-Based	Changes recorded externally and processed into TRS	Available in V7.0.2+	Better consistency (Out of date configurations, skipped resources, missing/extra requirements and out of date requirements), error detection (Better Serviceability), and validation (Detect/correct a wider range of problems)

What is the Delta Monitoring Task (DMT)?

The DMT is a Java process within ERM for log-based TRS - the purpose is to query for and process three kinds of change (in this order) which update the TRS.

1. Archive/Restore compensations
2. Durable log changes
3. Container recovery

• The DMT gives Administrators a way to determine how up to date TRS processing is.
• DMT can be monitored by Administrators in the Tracked Resource Set monitoring page which shows the Watermark (How up to date the TRS processing is – this was never visible in event-based TRS
• The DMT runs every 2 seconds (non-configurable).
• It is normal for the DMT to be a few seconds behind and can go further behind when there is a lot of data to process
  • So this isn't necessarily a problem until it continues for a long time, perhaps >15 minutes

Monitoring TRS Currency

To access the Tracked Resource set go to :

 https://<JazzServerName>/rm/admin/#action=com.ibm.rdm.fronting.server.web.trs

Displays how far behind TRS processing is (days/hours/minutes/seconds).

DMT Debug Logging Setup

To set debug Logging:

1. Navigate to /server/conf/rm/log4j2.xml
2. Make a backup of this file before editing it.
3. Add the following to the Loggers section?

Edit log4j2.xml:

<!-- DMT Debug Logging -->
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsDeltaMonitorTask" level="DEBUG"/>
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsChangeProcessor" level="DEBUG"/>
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.logbased.ContainerChangeProcessor" level="DEBUG"/>
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.feed.TrsChangeLogService" level="DEBUG"/>


<!--  As this is extremely verbose, only set this to Debug when/if waiting for pending change is present in the logs for a "long time", gather an hours log entries at DEBUG level and then set back to INFO.-->
<Logger name="com.ibm.rdm.fronting.server.services.trs.internal.logbased.InflightUpdates" level="DEBUG"/>


<!--  Use debug to get CSV-formatted metrics from TRS -->
<Logger name="com.ibm.rdm.fronting.server.services.trs.ITrsService.metrics" additivity="false" level="DEBUG">
<AppenderRef ref="trs_metrics"/></Logger>

DMT Debug Log Examples

• com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsDeltaMonitorTask

WARN: No watermark in the repository; compensations will remain pending until a watermark exists
DEBUG: DMT: begin/end compensating action
DEBUG: containerRecovery begin/end

• com.ibm.rdm.fronting.server.services.trs.internal.logbased.TrsChangeProcessor

DEBUG: DMT: Found %d version mapping changes in %s
DEBUG: Excluded container no longer being processed by AM.  Starting catch-up. %s
DEBUG: TrsChangeProcessor.commit: begin/end %d events from %s (normal)
DEBUG: TrsChangeProcessor.commit: %d compensating actions (compensation)

• com.ibm.rdm.fronting.server.services.trs.internal.logbased.ContainerChangeProcessor

2025-03-11T20:16:33,162+0200 [][TRS DMT] DEBUG ces.trs.internal.logbased.ContainerChangeProcessor - Processing configuration change CREATE on com.ibm.team.repository.service.internal.vvc.vvcmodel.impl.ConfigurationImpl@531d9de7 (stateId: [UUID _WlH7wDc2EfGOVqXvcnmz7w], itemId: [UUID _Vzm_QDc2EfGOVqXvcnmz7w], origin: <unset>, immutable: true) (contextId: [UUID _8lNyYNwSEd2pIJ5QVwgQGg], modified: 2025-03-10 14:43:20.828, workingCopy: <unset>) (predecessor: <unset>) (id: https://jazz.ibm.com/rm/cm/stream/_Wcm_QDc2EfGOVqXvcnmz7w, name: TEST_Configuration_XXX, description: null, creationTime: 2025-03-10 14:43:15.64, type: workspace, archived: false, context: null)
2025-03-11T20:16:33,162+0200 [][TRS DMT] DEBUG ces.trs.internal.logbased.ContainerChangeProcessor - Container ContainerChange [timestamp=2025-03-10 14:43:15.64, type=CREATE, item=com.ibm.team.repository.service.internal.vvc.vvcmodel.impl.ConfigurationImpl@531d9de7 (stateId: [UUID _XlH7wDc2EfGOVqXvcnmz7w], itemId: [UUID _Vzm_QDc2EfGOVqXvcnmz7w], origin: <unset>, immutable: true) (contextId: [UUID _8lNyYNwSEd2pIJ5QVwgQGg], modified: 2025-03-10 14:43:20.828, workingCopy: <unset>) (predecessor: <unset>) (id: https://jazz.ibm.com/rm/cm/stream/_Wcm_QDc2EfGOVqXvcnmz7w, name: TEST_Configuration_XXX, description: null, creationTime: 2025-03-10 14:43:15.64, type: workspace, archived: false, context: null)] CREATE request at:2025-03-10 14:43:15.64
2025-03-11T20:16:33,162+0200 [][TRS DMT] DEBUG ces.trs.internal.logbased.ContainerChangeProcessor - Configuration (stream=true) Wcm_QDc2EfGOVqXvcnmz7w create at: 2025-03-10 14:43:15.64
2025-03-11T20:16:33,162+0200 [][TRS DMT] DEBUG ces.trs.internal.logbased.ContainerChangeProcessor - Processed configuration change CREATE on com.ibm.team.repository.service.internal.vvc.vvcmodel.impl.ConfigurationImpl@531d9de7 (stateId: [UUID _WlH7wDc2EfGOVqXvcnmz7w], itemId: [UUID _Vzm_QDc2EfGOVqXvcnmz7w], origin: <unset>, immutable: true) (contextId: [UUID _8lNyYNwSEd2pIJ5QVwgQGg], modified: 2026-04-13 14:43:20.828, workingCopy: <unset>) (predecessor: <unset>) (id: https://jazz.ibm.com/rm/cm/stream/_Vzm_QDc2EfGOVqXvcnmz7w, name: EBS_L2_SYSP_Configuration_Project_OVERLANDAI_ORV_MK_120_DG-101103, description: null, creationTime: 2025-03-10 14:43:15.64, type: workspace, archived: false, context: null)
2025-03-11T20:16:33,162+0200 [][TRS DMT] DEBUG .services.trs.internal.logbased.TrsChangeProcessor - Item [_-4rQUGh7EfCrGMGVoeJ6BA._-5wOYWh7EfCrGMGVoeJ6BA] not analyzed because timestamp matches configuration timestamp : https://jazz.ibm.com/rm/cm/stream/_Wcm_QDc2EfGOVqXvcnmz7w.

• com.ibm.rdm.fronting.server.services.trs.internal.feed.TrsChangeLogService (This is extremely verbose as it logs each transaction the DMT is processing, "CREATION" ,"DELETION", "MODIFICATION","ARCHIVE" and "RESTORE".

2025-03-11T20:16:47,417+0200 [][RM-TRS Event Processing (serial)-1] DEBUG ver.services.trs.internal.feed.TrsChangeLogService - TRS 2.0 addChangeLogEvents( eventType=CREATION, url=https://jazz.ibm.com/rm/versionedResources/BI_JvLr0PzlEfClPdo9upNKPQ/state/_zY4msQJfEfGLk4Rz5jos_w/core/TX_JwprdfzlEfClPdo9upNKPQ/state/_AAqlcDc4EfGOVqXvcnmz7w, rationale = NORMAL, etag = null )
2025-03-11T20:16:47,417+0200 [][RM-TRS Event Processing (serial)-1] DEBUG ver.services.trs.internal.feed.TrsChangeLogService - TRS 2.0 addChangeLogEvents( eventType=MODIFICATION, url=https://jazz.ibm.com/rm/configSelections/stream/_s0vAltbxEeqkc-58C6IiSQ, rationale = NORMAL, etag = 1.4a5d757f, patch= { beforeETag=1.cb681fa9, beforeHashCode=-1734854246194175041, afterETag=1.4a5d757f, afterHashCode=-6117785062366859013, rdfPatch='

• com.ibm.rdm.fronting.server.services.trs.ITrsService.metrics

2025-03-11T04:26:40,767, watermark, 2025-03-11T11:26:32.911Z,1741692392911, 2025-03-11T11:26:35.097Z,1741692395097, 2025-03-11T11:56:32.911Z,1741694192911, 0,0, 2025-03-11T11:26:35.097Z,1741692395097, 0, 129, 5410
2025-03-11T04:26:42,939, watermark, 2025-03-11T11:26:35.097Z,1741692395097, 2025-03-11T11:26:42.814Z,1741692402814, 2025-03-11T11:56:35.097Z,1741694195097, 0,0, 2025-03-11T11:26:42.814Z,1741692402814, 0, 0, 94

DMT Performance Considerations

• Server load
• Large deliveries
• Database performance
• Archive/restore operations

Use IBM MustGather for diagnostics: IBM DOORS Next Generation MustGather

• Foundation Counters:

   https://<JazzServer>/rm/service/com.ibm.team.repository.service.internal.counters.ICounterContentService 

If there is a queue reported it can indicate load/wait times at the database level

• Resource Usage (Pools):

1. Check RM Statistics:
2. Gather Statistics:Navigate to https:///rm/debug
3. Click on Statistics

Identifying DMT Processing

• Archive/Restore: If the DMT is processing Archive/Restore requests the DMT page will display the ID of the configuration and the type of change (Archive/Restore)

• Cache folder takes up a lot of space under "\server\conf\rm\com.ibm.team.jfs.cache.service.internal": The reason for folder growth: *

Every TRS-relevant operation including artifact creation or modification, baseline archival, and stream operations generates TRS change events. In log-based TRS mode, the TRS Delta Monitoring Task (DMT) reads these events from a durable log in the database and materializes the TRS changelog pages in the on-disk cache directory.

The growth of the cache directory can therefore occur due to the following reasons:

1. Mass baseline archival: Bulk baseline archival operations can generate a significant volume of ARCHIVE-type TRS deletion events, resulting in a corresponding increase in cache files.
2. Cache file retention: Files get automatically removed when the last reader closes. (If the server crashes before this happens → the file is left on disk as an orphan.)
3. Files in this folder are safe to delete only when the server is fully stopped. They are not live database data only read/write buffers and read caches. Deleting them while the server is running risks corrupting an in-flight content serve or cache read.
   1. Let the JFSCacheScrubber handle it automatically on the next restart. Property: JFS Cache Scrubber Fixed Delay (seconds) If this DOORS Next Advanced property is set to -1, the cache scrubber will not be scheduled when the server restarts and orphaned files will not be cleaned up.

DMT Known issues:

List of Known issues that are known to affect DMT:

Known issue	Description	Fixed in/Scheduled for
DT501382	TRS Delta Monitor Task (DMT) can cause excessive heap usage when processing a large number of recent changes (152716)	Scheduled for TBD
DT471103	TRS etag mismatch between GET/Base and changelog event (151256)	Scheduled for V7.1 ifix013, v7.2 ifix005
DT469185	Unknown container replacement can result in blocking the TRS from updating resulting in TRS lag (151520)	Fixed in V7.1 ifix011, v7.2 ifix003 and V7.0.3 ifix023
DT468525	TRS DMT: registrations must be removed when the transaction completes (598358) See Flash on this DOORS Next Log Delta Monitoring Task (DMT) may stall resulting in processing delays (Affects Log based only)	Fixed in V7.1 ifix010, v7.2 ifix004 and V7.0.3 ifix023
DT468092	TRS DMT does not process compensation events promptly (151504)	Fixed in V7.1 ifix011, v7.2 ifix004 and V7.0.3 ifix023
DT453905	Tracked Resource Set(TRS) Delta Monitor task(DMT) lag resulting in poor query performance in DMT (150392)	Fixed in V7.1 ifix009 (Scheduled for V7.0.3 ifix024 and V7.2 ifix005)
DT454904	IChangeSetService.getEarliestPendingChangeSetCommitTime() returns out-of-date timestamps (591959)	Fixed in V7.0.3 ifix022, V7.1 ifix007, V7.2 GA
DT457527	TRS Archive Error : Unknown container replacement can result in blocking the TRS from updating resulting in stale data (150715)	Fixed in V7.0.3 ifix022, V7.1 ifix009, V7.2 ifix002

DMT Open Ideas:

Idea ID	Description	Status
ENGRMDN-I-1223	DOORS Next Delta Monitoring Task (DMT) should be able to be tracked outside of the UI and also alert Jazz admins if it falls behind	Future consideration

Related topics: Deployment web home, Deployment web home

External links:

• IBM