Dear Experts,

For the last 1 month, we have been receiving the "CRJAZ1413E Database connection is null. Check database status and availability" error in the logs very often and connection to RTC fails.
We have raised a PMR but we have no resolution as yet.
Of late, when we restart the server to overcome this issue as a workaround, we have also been receiving the below error. Could someone please help us in identifying the below issue and if there is any action we ought to do to resolve this asap ..

Error Message,


2011-08-10 12:57:32,277 WARN com.ibm.team.repository - CRJAZ0986I An exception occurred while trying to add some log entries to the feed.
com.ibm.team.repository.common.TeamRepositoryException: CRJAZ0987I Failed to create a change event for the log entry.
at com.ibm.team.repository.service.internal.LogChangeEventAppenderTask.saveChangeEventsForLogEntries(LogChangeEventAppenderTask.java:105)
at com.ibm.team.repository.service.internal.LogChangeEventAppenderTask.runTask(LogChangeEventAppenderTask.java:124)
at com.ibm.team.repository.service.async.AbstractAutoScheduledTask.executeTask(AbstractAutoScheduledTask.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at $Proxy115.executeTask(Unknown Source)
at com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.runTask(AsynchronousTaskRunner.java:118)
at com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.run(AsynchronousTaskRunner.java:92)
at java.lang.Thread.run(Thread.java:811)
Caused by:
com.ibm.team.repository.common.InternalRepositoryException: Deferred operation failed
at com.ibm.team.repository.service.internal.RdbRepositoryDataMediator.preCommitNotify(RdbRepositoryDataMediator.java:1561)
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase$WriteExclusiveTransaction.completeTransaction(RepositoryDatabase.java:666)
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase$Transaction.finishTransaction(RepositoryDatabase.java:500)
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase$1.run(RepositoryDatabase.java:293)
at com.ibm.team.repository.service.internal.rdb.ConnectionPoolService.withCurrentConnection(ConnectionPoolService.java:331)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at $Proxy22.withCurrentConnection(Unknown Source)
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase.runTransaction(RepositoryDatabase.java:288)
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase.runInTransaction(RepositoryDatabase.java:244)
at com.ibm.team.repository.service.internal.TransactionService.runInTransaction(TransactionService.java:79)
at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at $Proxy44.runInTransaction(Unknown Source)
at com.ibm.team.repository.service.internal.RepositoryItemService.saveItemInTxn(RepositoryItemService.java:1163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at $Proxy74.saveItemInTxn(Unknown Source)
at com.ibm.team.repository.service.internal.ComponentRepositoryItemService.saveItemInTxn(ComponentRepositoryItemService.java:557)
at com.ibm.team.repository.service.internal.ChangeEventService.save(ChangeEventService.java:149)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at $Proxy80.save(Unknown Source)
at com.ibm.team.repository.service.internal.LogChangeEventAppenderTask.saveChangeEventsForLogEntries(LogChangeEventAppenderTask.java:97)
... 13 more
Caused by:
com.ibm.team.repository.common.TeamRepositoryException: deferred save failed
at com.ibm.team.repository.service.internal.RdbRepositoryDataMediator.saveItemInTxn(RdbRepositoryDataMediator.java:490)
at com.ibm.team.repository.service.internal.RepositoryItemService.doSaveItemInTxn(RepositoryItemService.java:1191)
at com.ibm.team.repository.service.internal.RepositoryItemService.access$11(RepositoryItemService.java:1184)
at com.ibm.team.repository.service.internal.RepositoryItemService$14.run(RepositoryItemService.java:1165)
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase$Transaction.run(RepositoryDatabase.java:466)
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase$1.run(RepositoryDatabase.java:292)
... 51 more
Caused by:
com.ibm.team.repository.common.InternalRepositoryException: CRJAZ0447I SQL statement execution failed.Integrity constraint violation
SQL: INSERT INTO JAZZDBUSER.REPOSITORY_CHANGE_EVENT (STATE_ID, ITEM_ID, CONTEXT_ID, MODIFIED, MODIFIED_BY_ITEM_ID, EVENT_OWNER_NS, EVENT_TIME, EVENT_AUTHOR_ITEM_ID, EVENT_CATEGORY, EVENT_PRIORITY, EVENT_PROCESS_AREA_ITEM_ID, JZ_EVENT_PROCESS_AREA_DSCRMNTR, EVENT_EXPIRATION, ITEM_ITEM_ID, JZ_ITEM_DISCRIMINATOR) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
SQL Exception #1
SQL Message: ORA-00001: unique constraint (JAZZDBUSER.RPSTR_RPSTR_CHNG_VNT_PK) violated

SQL State: 23000
Error Code: 1
Class: com.ibm.team.repository.service.internal.dataaccess.write2.Row
Item Handle: com.ibm.team.repository.common.model.impl.ChangeEventImpl@32663266 (stateId: , itemId: , origin: <unset>, immutable: true) (contextId: , modified: 2011-08-10 12:57:32.12, workingCopy: false) (predecessor: <unset>) (eventState: null, eventOwnerNS: , eventTime: 2011-08-10 12:32:37.642, eventCategory: SystemLog, eventTitle: WARN - Error querying for contributor: asiqfareeth, eventDescription: Error querying for contributor: asiqfareeth. CRJAZ1170I The request was made by user "asiqfareeth" from "10.92.2.31". CRJAZ1166I The stack trace hash is D72AE40F813208F0CDD3DE97244250226C5A3073.

<h6>Throwable</h6>
com.ibm.team.repository.common.TeamRepositoryException: Error querying for contributor: asiqfareeth<br>
at com.ibm.team.repository.service.internal.ContributorCache.executeQuery(ContributorCache.java:203)<br>
at com.ibm.team.repository.service.internal.ContributorCache$ContributorCacheHelper.computeValue(ContributorCache.java:226)<br>
at com.ibm.team.repository.service.internal.ContributorCache$ContributorCacheHelper.computeValue(ContributorCache.java:1)<br>
at com.ibm.team.repository.service.internal.rdb.TransactionalCache$Compute1AndCache.compute(TransactionalCache.java:1895)<br>
at com.ibm.team.repository.service.internal.rdb.TransactionalCache.internalGet(TransactionalCache.java:583)<br>
at com.ibm.team.repository.service.internal.rdb.TransactionalCache.getWithTask(TransactionalCache.java:369)<br>
at com.ibm.team.repository.service.internal.rdb.TransactionalCache.get(TransactionalCache.java:353)<br>
at com.ibm.team.repository.service.internal.rdb.TransactionalCache.get(TransactionalCache.java:1)<br>
at com.ibm.team.repository.service.internal.ContributorCache.get(ContributorCache.java:82)<br>
at com.ibm.team.repository.service.internal.ContributorService.getInfoFromCache(ContributorService.java:344)<br>
at com.ibm.team.repository.service.internal.ContributorService.fetchContributorItemIdByUserId(ContributorService.java:332)<br>
at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source)<br>
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br>
at java.lang.reflect.Method.invoke(Method.java:618)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)<br>
at $Proxy110.fetchContributorItemIdByUserId(Unknown Source)<br>
at com.ibm.team.repository.servlet.TeamServerServlet.getContributor(TeamServerServlet.java:87)<br>
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:1747)<br>
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:1642)<br>
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1555)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)<br>
at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:180)<br>
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)<br>
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)<br>
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:75)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)<br>
at org.eclipse.equinox.servletbridge.BridgeServlet.service(BridgeServlet.java:121)<br>
at com.ibm.team.repository.server.servletbridge.JazzServlet.service(JazzServlet.java:54)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)<br>
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)<br>
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)<br>
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)<br>
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)<br>
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)<br>
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)<br>
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)<br>
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)<br>
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)<br>
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:873)<br>
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)<br>
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)<br>
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)<br>
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)<br>
at java.lang.Thread.run(Thread.java:811)<br>
Caused by: <br>
com.ibm.team.repository.common.InternalRepositoryException: CRJAZ1413E Database connection is null. Check database status and availability.<br>
at com.ibm.team.repository.service.internal.rdb.JDBCConnectionPool.checkConnection(JDBCConnectionPool.java:122)<br>
at com.ibm.team.repository.service.internal.rdb.JDBCConnectionPool.getConnection(JDBCConnectionPool.java:94)<br>
at com.ibm.team.repository.service.internal.rdb.ConnectionPoolService.withCurrentConnection(ConnectionPoolService.java:318)<br>
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)<br>
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br>
at java.lang.reflect.Method.invoke(Method.java:618)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)<br>
at $Proxy22.withCurrentConnection(Unknown Source)<br>
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase.runTransaction(RepositoryDatabase.java:288)<br>
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase.runInTransaction(RepositoryDatabase.java:244)<br>
at com.ibm.team.repository.service.internal.TransactionService.runInTransaction(TransactionService.java:79)<br>
at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)<br>
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br>
at java.lang.reflect.Method.invoke(Method.java:618)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)<br>
at $Proxy44.runInTransaction(Unknown Source)<br>
at com.ibm.team.repository.service.internal.QueryService.doQueryData(QueryService.java:1054)<br>
at com.ibm.team.repository.service.internal.QueryService.queryData(QueryService.java:1041)<br>
at sun.reflect.GeneratedMethodAccessor119.invoke(Unknown Source)<br>
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br>
at java.lang.reflect.Method.invoke(Method.java:618)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)<br>
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)<br>
at $Proxy59.queryData(Unknown Source)<br>
at com.ibm.team.repository.service.internal.SecureQueryService.queryData(SecureQueryService.java:140)<br>
at com.ibm.team.repository.service.internal.ContributorCache.executeQuery(ContributorCache.java:195)<br>
... 45 more<br>

<h6>Location</h6>
com.ibm.team.repository.servlet.AbstractTeamServerServlet.logWarningWithClientDetails(AbstractTeamServerServlet.java:1293), eventPriority: , eventExpiration: 2011-08-13 12:57:32.029)
at com.ibm.team.repository.service.internal.dataaccess.BatchingRowReceiver.createIRE(BatchingRowReceiver.java:417)
at com.ibm.team.repository.service.internal.dataaccess.BatchingRowReceiver.createIRE(BatchingRowReceiver.java:426)
at com.ibm.team.repository.service.internal.dataaccess.write2.Row.handleExecuteException(Row.java:96)
at com.ibm.team.repository.service.internal.dataaccess.BatchingRowReceiver.simulatedBatch(BatchingRowReceiver.java:396)
at com.ibm.team.repository.service.internal.dataaccess.BatchingRowReceiver.executeBatch(BatchingRowReceiver.java:269)
at com.ibm.team.repository.service.internal.dataaccess.BatchingRowReceiver.execute(BatchingRowReceiver.java:196)
at com.ibm.team.repository.service.internal.dataaccess.IRowReceiver$IRowReceiverLocator.executePendingChanges(IRowReceiver.java:140)
at com.ibm.team.repository.service.internal.dataaccess.IRowReceiver$IRowReceiverLocator.executePendingChanges(IRowReceiver.java:118)
at com.ibm.team.repository.service.internal.RdbRepositoryDataMediator.preCommitNotify(RdbRepositoryDataMediator.java:1534)
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase$WriteExclusiveTransaction.completeTransaction(RepositoryDatabase.java:666)
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase$Transaction.finishTransaction(RepositoryDatabase.java:500)
at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase$1.run(RepositoryDatabase.java:293)
... 51 more
Caused by:
java.sql.SQLException: ORA-00001: unique constraint (JAZZDBUSER.RPSTR_RPSTR_CHNG_VNT_PK) violated

at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:745)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:219)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:970)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1190)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3370)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3454)
at com.ibm.team.repository.service.internal.db.jdbcwrappers.stat.PreparedStatementStatWrapper.executeUpdate(PreparedStatementStatWrapper.java:62)
at com.ibm.team.repository.service.internal.db.jdbcwrappers.errlog.PreparedStatementErrLogWrapper.executeUpdate(PreparedStatementErrLogWrapper.java:86)
at com.ibm.team.repository.service.internal.db.jdbcwrappers.leak.PreparedStatementLeakWrapper.executeUpdate(PreparedStatementLeakWrapper.java:58)
at com.ibm.team.repository.service.internal.dataaccess.BatchingRowReceiver.simulatedBatch(BatchingRowReceiver.java:388)