JTS and CCM Component/Services Fail to Load
I hope someone can help me out - I've spent nearly two days on this.
ENV:
AIX 5.3 RTC 3.0.1
PROBLEM SUMMARY:
JTS and CCM components are not loading.
The JTS starts, however none of the services are available
BACKGROUND:
Server has been running fine for 2 months
I noticed online backups had stopped Jan 6, due to internal GSA filesystem issue, in order to restart GSA, I rebooted, probably not taking the team server down correctly, of course.
# repotools-jts.sh -verify
ACTIONS TAKEN:
I cannot find any information on this
Package URI == com.ibm.team.repository.auth
What else is there to reinstall, remove, etc?
Assuming this Database Version (4) is in the db2 database, is it possible that this was invalid prior to the reboot, however would not appear until the next startup of the server?
This is a production server, any help is greatly appreciated!!!
AIX 5.3 RTC 3.0.1
JTS and CCM components are not loading.
The JTS starts, however none of the services are available
Server has been running fine for 2 months
I noticed online backups had stopped Jan 6, due to internal GSA filesystem issue, in order to restart GSA, I rebooted, probably not taking the team server down correctly, of course.
# repotools-jts.sh -verify
Repo Tools
Provisioning using "./conf/jts/provision_profiles".
Jazz Foundation - Core Libraries, Version 3.0.1 (RJF-I20110602-0017)
CRJAZ1363I Loading the configuration from "file:conf/jts/teamserver.properties".
CRJAZ1779I This server is configured as a JTS.
CRJAZ1971I The server is configured with:
Public URI: "https://myserver.ibm.com:9443/jts"
CRJAZ1365I The server is attempting to connect to the following database: "//myserver.ibm.com:60000/JTS:user=xxxxxxxx;password=xxxxxxxx;"
CRJAZ1364I The connection to the following database was successful:
Db Product Name: DB2/AIX64
Db Product Version: SQL09074
Db URL: jdbc:db2://myserver.ibm.com:60000/JTS:user=xxxxxxxx;password=xxxxxxxx;
Jdbc Driver Name: IBM DB2 JDBC Universal Driver Architecture
Jdbc Driver Version: 3.57.82
CRJAZ8192I: Fulltext Index Location: /home/rtc/rtcCurrent/silent-install/JazzTeamServer/server/conf/jts/indices/workitemindex/fulltext_index
CRJAZ8190I: The fulltext index should point to an absolute directory but is currently set to 'conf/jts/indices/workitemindex'
CRJAZ1042I The component has a server version mismatch between the installed code and the database.
**-> Package URI == com.ibm.team.repository.auth
Installed Version == 3
Database Version == 4
CRJAZ1049I A runtime exception occurred during repository startup.
CRJAZ1093I The following service class was not activated: "com.ibm.team.repository.service.internal.RepositoryItemService"
CRJAZ1141I Cannot get the service "com.ibm.team.repository.service.IRepositoryItemService".
CRJAZ0688I Unable to initialize service "repository item service".
CRJAZ1161I Failed to get "com.ibm.team.repository.service.internal.verifiers.RepositoryVerifier" implementation.
CRJAZ1161I Failed to get "com.ibm.team.dashboard.service.internal.migration.RepositoryVerifier" implementation.
CRJAZ1161I Failed to get "com.ibm.team.process.internal.service.migration.ProcessMigrationVerifier" implementation.
CRJAZ1161I Failed to get "com.ibm.team.reports.service.internal.ReportVerifier" implementation.
Verifying the repository contents using verification level 1.
Verifying items present in the "com.ibm.team.repository.systemtables" component.
- -Removed all cache files I could find
-Redeployed ccm.war, jts.war, even admin.war
-Restored CCM, JTS and DW from backups made Jan 6
-Ran OnlineVerifyTool as directed: http://jazz.net/wiki/bin/view/Main/OnlineVerifyTool
-repotools-xxx.sh -resetRepoLockId (xxx=jts and ccm)
-repotools-xxx.sh -addTables noPrompt -clean
I cannot find any information on this
What else is there to reinstall, remove, etc?
Assuming this Database Version (4) is in the db2 database, is it possible that this was invalid prior to the reboot, however would not appear until the next startup of the server?
This is a production server, any help is greatly appreciated!!!
3 answers
There is version skew between the code you are running and the format of your database. The code version is older than the database. Your database was created with code that include version 4 of the "com.ibm.team.repository.auth" component, but now your server is running code where the version is 3.
So somehow you either installed an older version of the server software, or pointed it to a database created with a newer version. At least, that's what the error messages are indicating.
So somehow you either installed an older version of the server software, or pointed it to a database created with a newer version. At least, that's what the error messages are indicating.
There is version skew between the code you are running and the format of your database. The code version is older than the database. Your database was created with code that include version 4 of the "com.ibm.team.repository.auth" component, but now your server is running code where the version is 3.
So somehow you either installed an older version of the server software, or pointed it to a database created with a newer version. At least, that's what the error messages are indicating.
Thanks for the quick response, jrvasta. The problem is - I've done no updates nor changed databases. This has been running for two months with no problems. I cannot find any information about this "com.ibm.team.repository.auth" component, either so I don't even know how to correct this, or how I could have created a newer version database without knowing it.
Additionally - what is the version referring to? RTC Version 4? Or a code level for that package? (which I cannot seem to locate, any tips on that would be helpful)
While I would really like to understand what this package is, and how the database could have been updated under the covers, I was able to get this restored and running, so am posting my steps for other readers to *enjoy* if they have similar problems
WORKAROUND:
I effectively reinstalled the team server by copying over the original install for 3.0.1.1
The way the installs were setup for updates on our server is:
Which leaves a filesystem rtcNNN which can be removed when the next version is available, or in this case, if we need to replace the rtcCurrent
For testing purposes, I mounted this /rtc3011/silent-install over /home/rtc/rtcCurrent/silent-install
Had to reset the repository lock id of the database
Then restarted the RTC server:
server.startup
That restored the server so I proceeded to replace the entire installation directory.
I unmounted the /home/rtc/rtcCurrent/silent-install and removed it, copying over the working installation
cp -R /rtc3011/silent-install /home/rtc/rtcCurrent/silent-install
I effectively reinstalled the team server by copying over the original install for 3.0.1.1
The way the installs were setup for updates on our server is:
- Create a filesystem for the new version /rtc3011 and install the new version
Perform the RTC instructions as advised by Jazz.
Copy this to /home/rtc/rtcCurrent
Which leaves a filesystem rtcNNN which can be removed when the next version is available, or in this case, if we need to replace the rtcCurrent
For testing purposes, I mounted this /rtc3011/silent-install over /home/rtc/rtcCurrent/silent-install
Had to reset the repository lock id of the database
- cd /rtc/rtcCurrent/silent-install/JazzTeamServer/server
repotools-jts.sh -resetRepoLockId
repotools-ccm.sh -resetRepoLockId
Then restarted the RTC server:
server.startup
That restored the server so I proceeded to replace the entire installation directory.
I unmounted the /home/rtc/rtcCurrent/silent-install and removed it, copying over the working installation
cp -R /rtc3011/silent-install /home/rtc/rtcCurrent/silent-install