Troubleshooting - ELM Server Rename

Authors: Chao Wang, Kot Tontranakwong, Susan Wu
Build basis: Build basis: ELM 6.x and 7.x and later

This page provides information on how to troubleshoot server rename issues in IBM Engineering Lifecycle Management applications (ELM).

Getting Started

Need to know:

• At what stage in the rename process the failure occurs. Depending on where the failure occurs, the logs to collect and review can be different.
• What files to gather. In most cases, ISADC includes all the files that are needed for troubleshooting a rename issue.
• What to look for in those files
• How to read the logs
  

At what stage the failure occurs?

1. Generate / Verify URL mappings
2. Import URL mappings
3. JTS rename status
4. Link Validation
5. Finalize rename
6. After a successful rename

Point of Failure

1. Generate / Verify URL Mappings

During the run of the repotools-jts generateURLMapping command, JTS contacts all registered ELM applications to get all possible URLs, then it generates a mapping file and an additionalURLs file. After that edit the generated mapping and additinoalURLs files, check Mappings File section in this wiki pag for details of what to watch for in preparing for the mapping file. Once done run repotools-jts verifyURLMappings command to verify mapping file is valid. These are online operations, therefore, all applications must be running.

Files to gather:

• Repotools-jts_generateURLMappings.log
• Repotools-jts_verifyURLMappings.log
• Application logs (jts.log, ccm.log, etc.)

What to look for:

• In repotools-jts_generateURLMapping.log, look for the following line to see what time it starts to collect URL information and what time it fails.

  2025-04-04T14:51:10,220+0000 CRJAZ2233I The server at https://<server:port>/jts/urlMappingInformation is being contacted to collect URL information.
  2025-04-04T14:56:10,227+0000 CRJAZ2942E A request from this server to another server could not be completed. The other server returned a 504 HTTP error with this error text: Gateway Time-out.
  com.ibm.team.repository.common.transport.TeamServiceException: CRJAZ2942E A request from this server to another server could not be completed. The other server returned a 504 HTTP error with this error text: Gateway Time-out.
  

• Prior running this command, check the URL Alias under jts advanced property is empty or contain valid value
• If the failure occurs immediately, get the timestamp when the error occurs and search in jts.log to find out which application it was trying to get data from and fail.
• If the failure occurs at least 5+ minutes and the error indicates some time-out issue, check the following technotes for how to extend the timeout settings:
  Technotes:
  • • 290539: ELM repotools command generateURLMappings or verifyURLMappings command fails with CRJAZ0099E error
    • 6590385: Troubleshooting SocketTimeoutException received when running repotools-jts GenerateURLMappings
    • 7066586: Performing Server Rename in ELM results in Unhandled Exception

2. Import URL Mappings

Prior running repotools-jts importURLMapping command, review the Mappings File section in this wiki page, ensure that the mapping file is good, and that the verifyURLMappings log reports no error.
The repotools-jts importURLMapping command performs the following tasks:

• Populate the following tables in the jts repository database:
  1 REPOSITORY.QUERYABLE_JPI_MAPPING
  2 REPOSITORY.JPI_MAPPING
  3 REPOSITORY.MAPPING_EVENT
  4 REPOSITORY.MAPPING_EVENT_URL_CHANGES tables
  
• Generates the .mappingevent file under conf/jts folder which contains the following two URLs for ELM applications to contact jts with. This mapping event file needs to be manually copied over to other elm applications in distributed environment if network drives are not mapped:
  1 https://targetserver.com:9443/jts/urlMappings/prefixes >> retrieve the mapping event
  2 https://targetserver.com:9443/jts/serverRenameStatus >> retrieve server rename status
  
• Update the jts.url and web.server.url in jts teamserver.properties file
• Attempts to rename URLs in the Data Warehouse (DW) database

Files to gather:

• Repotools-jts_importURLMappings.log
• Repotools-jts_verifyURLMappings.log:

What to look for:
repotools-jts_importURLMappings.log:

• Check for errors
• Look in the command and see which mapping file is used

repotools-jts_verifyURLMappings.log:

• Confirm the command is run against the mapping file used in to import and that there is no error.

3. JTS Rename Status

In ELM versions (and iFix versions) prior to 7.2, 7.1 iFix004, and 7.0.3 iFix015, the progress of the rename status is not written to JTS log. As a workaround, you can manually enable the logging by adding the following log4j logger to JTS.

 <Logger name="com.ibm.team.repository.service.mapping.internal.ServerRenameOverallStatusRestService" level="DEBUG"/> 

Related Technote and Defect:

• 7058385: Jazz Team Server does not report server rename status within the jts.log
• https://jazz.net/jazz/resource/itemName/com.ibm.team.workitem.WorkItem/582431

Files to gather:

• Repotools-jts_importURLMappings.log
• Application logs (jts.log, ccm.log, etc.)

What to look for:
repotools-jts_importURLMappings.log

• Check the log to see what time the rename was run

jts.log

• Check jts.log to see what time JTS polls and stops polling other applications for rename status

   2025-01-31T11:59:58,781+0100 [ Thread-51] [ INFO] ping.internal.ServerRenameOverallStatusRestService - Server rename details:Application polling is complete 

• Look for the error to know which applications fail and at what time.

   [ INFO] ping.internal.ServerRenameOverallStatusRestService - Server rename details:Unable to contact https://<server:port>/ccm/rootservices : java.net.SocketTimeoutException: connect timed out 

The corresponding applications’ log (eg, ccm.log, qm.log, etc)

• Check whether the application completes the rename process or fails with an error

   2025-01-31T12:30:51,717+0100 [][com.ibm.team.repository.service.mapping.internal.ServerRenameService] [ INFO] sitory.service.mapping.internal.ServerRenameHelper - CRJAZ2327I Server rename completed successfully for the application in 20213 ms. 

• State of the application when JTS was polling for status

   2025-01-31T12:29:25,468+0100 [com.ibm.team.repository.service.mapping.internal.ServerRenameService] [ INFO] sitory.service.mapping.internal.ServerRenameHelper - CRJAZ2366I Processing server rename mapping events... 

Explanation

• In the above example, ELM application server starts sometimes before 2025-01-31T11:59.
• JTS polls for application rename status and stops polling at 2025-01-31T11:59.
• JTS reports connect timed out on CCM rootservices URL. So, the log to review next is ccm.log.
• CCM is started at the same time as JTS, but the application is not fully loaded until around 2025-01-31T12:29:25.
• CCM starts to process the rename event after it fully loads up at 2025-01-31T12:29:25.
• CCM completes the rename at 2025-01-31T12:30:51.
• In this case, JTS reports CCM rename status as 'failed' because CCM is not available when JTS polls for rename status. However, the rename on CCM completes successfully afterward. In this case, the solution is to restart JTS so it starts to poll for rename status again. No action is needed on CCM.

4. Link Validation

• During the validation step, each application fetches a list of projects. If there is a problem loading the project list, look in JTS and the corresponding application logs to understand why.​
• If it shows many un-renamed links, find out why. Most of the time, it’s related to the targets in the mappings file.
  Technotes:
  • 7177052: Error 400 when retrieving a list of project areas during the verification process after a server rename

5. Finalize rename

• Most of the time, the rename process cannot be finalized because of some other issues in the rename process.

6. After a successful rename

• There are actions required as post-rename tasks. See Impact of Server Rename on ELM.

Mappings File

What to consider when you review the mapping files:

• Need Context Root
• Duplicated source or target URLs are not allowed
• URLs With No Port (Default Port 443)
• Include applications that are not part of the rename, but need to be renamed. For example, LQE, LDX, DWA, etc.
• URL should not be too specific
• Ensure Target URLs Are Valid
• Use Of IP Address In The Public URI?
• Target URLs must be in all lowercase letters

Need Context Root

• Always include the context when running server rename for the ELM applications. For non-ELM application URLs, it is possible that the URL does not have a context root.

  source=https://source.fyre.ibm.com:9443/<context>
  target=https://target.fyre.ibm.com:9443/<context>

Duplicated source or target URL is not allowed.

• All the mappings in the file must be unique. No multiple source to the same target.

  X X X - Example of bad mappings - X X X
  source=https://source.fyre.ibm.com/ccm
  target=https://target.fyre.ibm.com/ccm
  
  source=https://source.fyre.ibm.com:443/ccm
  target=https://target.fyre.ibm.com/ccm

URLs With No Port (Default Port 443)

• URLs with default port(443) are always in pairs.​

  source=https://c85616v1.fyre.ibm.com:443/jts
  target=https://testrename.fyre.ibm.com:443/jts
  
  source=https://c85616v1.fyre.ibm.com/jts
  target=https://testrename.fyre.ibm.com/jts

• URLs with explicit port are not required to be in pairs:

  source=https://c85616v1.fyre.ibm.com:9443/jts
  target=https://testrename.fyre.ibm.com:9443/jts

Include applications that are not part of the rename, but need to be renamed.

• Include applications that need to be renamed, even if they do not participate in the rename process. For example, LQE, LDX, DWA, Other third party URLs, etc.
• For LQE and LDX, generate the mapping file before unregistering LQE and LDX.

URL should not be too specific

• The source and target URLs should be specific enough, but not too specific that some artifacts are not renamed.
• The following URL is too specific, it only renames the root services URL:

  source=https://<third party app url>/rest/oslc/1.0/rootservices
  target=https://<new third party app url>/rest/oslc/1.0/rootservices 

Ensure Target URLs Are Valid

• Ensure the target URLs are available before starting the online portion of server rename.
• Typical issues are caused by:
  • Port
  • Context root
  • Hosts file
    This results in rename failure due to the URL not being available.

Use Of IP Address In The Public URI?

• Strongly discouraged due to:
  • IP addresses are not flexible
  • Simple hardware migration requires server rename
• Best practice:
  • Configure a reverse proxy(IHS) server

See Mapping File for Server Rename

Common Issues

URL Length

• This can happen when the target URL is longer than the source URL.
• Error:

  2023-01-19T15:44:10,286+0000 [com.ibm.team.repository.service.mapping.internal.ServerRenameService] ERROR sitory.service.mapping.internal.ServerRenameHelper - [a3519e8d] CRJAZ2359E An error occurred while processing the server rename mapping event "<rename event id>".
  com.ibm.team.repository.common.InternalRepositoryException: CRJAZ0329E The database query could not be completed. String data, right truncation
  
  SQL: Insert into {REPOSITORY.QUERYABLE_JPI_MAPPING} ({JPI}, {URL}) select distinct t1.URI, TRIM((t2.URL || substr(t1.URI, length(t2.JPI)+1,length(t1.URI)-length(t2.JPI)))) from MARKERS.MARKER t1,REPOSITORY.JPI_MAPPING t2
  where t1.URI is not null and t1.URI <> '' and t2.JPI = (select t3.JPI from REPOSITORY.JPI_MAPPING t3 where ((locate(t3.JPI || '/' ,t1.URI,1) = 1) or (t1.URI = t3.JPI)) order by length(t3.JPI) desc fetch first 1 rows only ) 
  and not exists (select 1 from {REPOSITORY.QUERYABLE_JPI_MAPPING M} where {M.JPI}=t1.URI)
  
  SQL Exception #1
  SQL Message: DB2 SQL Error: SQLCODE=-433, SQLSTATE=22001, SQLERRMC=https://server/qm/web/console/..., DRIVER=4.26.14
  SQL State: 22001
  Error Code: -433
  Error: Target URL length in mapping file is too long. Keep Target URL length shorter. Please refer "https://jazz.net/library/article/1165" for workarounds.
  ...
  Caused by: com.ibm.db2.jcc.am.SqlDataException: DB2 SQL Error: SQLCODE=-433, SQLSTATE=22001, SQLERRMC=https://server/qm/web/console/..., DRIVER=4.26.14 

• Workaround available: article 1165: workaround

JTS Failed To Poll The Correct Application Status​

• When JTS reports rename failure for an application, check the application's log file to find out its rename status. If you see a successful rename message below in the application log, you can restart just the JTS server for JTS to poll for the rename status again. It should then pick up the correct status and report a complete rename for the application.

  2025-01-17T12:10:11,912-0800 [com.ibm.team.repository.service.mapping.internal.ServerRenameService] INFO sitory.service.mapping.internal.ServerRenameHelper - CRJAZ2327I Server rename completed successfully for the application in 7376 ms.

ELM with Jazz Authorization Server

• If Jazz Authorization Server is configured, make sure to follow the server rename addendum steps:
  • ServerRenameAddendum
• Ensure all applications are installed with JSA enabled

  <ELM installation directory>\server\conf\ccm\provision_profiles\DISABLED 

During the URL validation steps, some URLs are not renamed (validation steps)

• Review the URL mappings in the mappings file:
  • The URLs in the mappings file are too specific
  • The URLs are not included in the mappings file
• Verifying URLs and links after a server rename

Q&A

Can I rename only CCM and not JTS and other apps?

• Yes, although CCM is the only application being renamed, the process to copy over the .mappingEvent file to all the applications remain the same.

How to get rename history?

• Repository tools command to export the URL mapping history of a server

  repotools-jts -exportServerRenameURLMappingHistory toFile=history.txt

How To Find The Current Mapping Event ID?
Ways to get the mapping event ID:

• Run the exportServerRenameURLMappingHistory command
• Once you start the server, the mapping event id is also in the application logs.

  2025-04-09T19:30:59,371-0700 [com.ibm.team.repository.service.mapping.internal.ServerRenameService]  INFO sitory.service.mapping.internal.ServerRenameHelper  - CRJAZ2364I Processing mapping event "_r5VlwBWrEfCrKbM_OMY9yg"…

• SQL query:
  • While all application have this table, the rename status code(APP_RENAME_STATUS) is only updated in JTS. Other applications will show 0.

    SELECT * FROM REPOSITORY.MAPPING_EVENT

Meaning of app_rename_status (0, 2, 5) in the repository.mapping_event table:

• In the JTS REPOSITORY.MAPPING_EVENT table, what does each "APP_RENAME_STATUS" code(0~5) mean?

  APP_RENAME_STATUS:
  0 - Not started - This is the state after the importURLMapping command is run
  1 - In progress
  2 - Completed - This is the state after finalizing the rename
  3 - Error
  4 - Validation
  5 - ReadOnly - This is the state after the rename status page reports success (all applications)

Related topics: Deployment web home, Deployment web home

External links:

• IBM

Additional contributors: TWikiUser, TWikiUser