RTC's email notification stopped responding

Hi guys,

ISSUE:

RTC's mail notification service has stopped working.

There are Java out of memory errors in Jazz.log.

What we tested:

1. Sent and received an email using mailx on command prompt from the AIX Server hosting RTC .

2. Verified settings are correct.

Can't Explain:

1. I'm able to send Team invites to users and receive them as CC but the user( configured for mail settings) isn't updating team members on WI or other changes.Not sure how they are related as I can send Team Invite email as an RTC Admin


Errors:

1.First appearance of error starts as below

2011-10-11 02:25:39,274 ERROR com.ibm.team.workitem - CRJAZ0852I An exception occurred while executing the task "n
otification.mail", contributed by component "com.ibm.team.workitem". The task will be removed from the schedule.
java.lang.OutOfMemoryError
at
java.lang.Object
.
clone
(Native Method)
at
com.ibm.icu.util.UResourceBundle$ResourceCacheKey
.
clone
(
UResourceBundle.java
:
4
1
1
)
at
(
ICUResourceBundle.java

at
com.ibm.team.workitem.service.internal.save.notify.ChangeEventMailNotifier
.
getFormattedTimestamp
(
ChangeEventMailNotifier.java
:
1
0
5
1
)
at
com.ibm.team.workitem.service.internal.save.notify.ChangeEventMailNotifier
.
createWorkItemChangedMail
(
ChangeEventMailNotifier.java
:
5
1
7
)
at
com.ibm.team.workitem.service.internal.save.notify.ChangeEventMailNotifier
.
processOneChange
(
ChangeEventMailNotifier.java
:
org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord
.
invoke
(
ExportProxyServiceRecord.java
:
3
7
0
)
at
org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord
.
access$0
(
ExportProxyServiceRecord.java
:
3
5
6
)
at
org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler
.
invoke
(
ExportProxyServiceRecord.java
:
5
6
)
at
$Proxy256
.
executeTask
(Unknown Source)
at
com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner
.
runTask
(
AsynchronousTaskRunner.java
at
com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner
.
run
(
AsynchronousTaskRunner.java
:
9
2
)
at
java.lang.Thread
.
run
(
Thread.java
:
7
3

2.11-10-11 02:25:45,953 WARN LeakWrapperLogger - Aborting transaction Connection #141380943 has 1 open statem
ents
java.lang.Exception: Stack Trace
at com.ibm.team.repository.service.internal.db.jdbcwrappers.leak.AbstractJDBCLeakWrapper.<init>(AbstractJDBCLeakWrapper.java:33)
at com.ibm.team.repository.service.internal.db.jdbcwrappers.leak.ConnectionLeakWrapper.<init>(ConnectionLeakWrapper.java:52)
at com.ibm.team.repository.service.internal.db.jdbcwrappers.JDBC3WrapperFactory.createLeakWrapper(JDBC3WrapperFactory.java:31)
at com.ibm.team.repository.service.internal.rdb.ConnectionFactory.createConnection(ConnectionFactory.java:128)
at com.ibm.team.repository.service.internal.rdb.JDBCConnectionPool.createEntry(JDBCConnectionPool.java:57)
at com.ibm.team.repository.service.internal.rdb.AbstractPool.getEntry(AbstractPool.java:199)
at com.ibm.team.repository.service.internal.rdb.JDBCConne
3.2011-10-11 02:25:45,498 ERROR com.ibm.team.process - CRJAZ0852I An exception occurred while executing the task "P
rocessChangeEvents", contributed by component "com.ibm.team.process". The task will be removed from the schedule.
java.lang.OutOfMemoryError
at
java.lang.Throwable
.
fillInStackTrace
2011-10-11 02:25:46,319 WARN LeakWrapperLogger - Aborting transaction Connection #22225552 has 1 open stateme
nts
java.lang.Exception: Stack Trace
at com.ibm.team.repository.service.internal.db.jdbcwrappers.leak.AbstractJDBCLeakWrapper.<init>(AbstractJDBCLeakWrapper.java:33)
at com.ibm.team.repository.service.internal.db.jdbcwrappers.leak.ConnectionLeakWrapper.<init>(ConnectionLeakWrapper.java:52)
at com.ibm.team.repository.service.internal.db.jdbcwrappers.JDBC3WrapperFactory.createLeakWrapper(JDBC3WrapperFactory.java:31)
at com.ibm.team.repository.service.internal.rdb.ConnectionFactory.createConnection(ConnectionFactory.java:128)
at com.ibm.team.repo
011-10-11 02:25:46,918 WARN LeakWrapperLogger - Aborting transaction Connection #22225530 has 1 open stateme
nts
java.lang.Exception: Stack Trace

followed by DB2 Errors:

2011-10-12 05:01:45,132 ERROR ository.service.internal.RdbRepositoryDataMediator - InternalRepositoryException processing query
com.ibm.team.repository.common.InternalRepositoryException: unable to build temp table
at com.ibm.team.repository.service.internal.db.util.InListValueTempTableManager.build(InListValueTempTableManager.java:106)
at com.ibm.team.repository.service.internal.RdbRepositoryDataMediator.doQueryData(RdbRepositoryDataMediator.java:913)
at com.ibm.team.repository.service.internal.RdbRepositoryDataMediator.queryData(RdbRepositoryDataMediator.java:884)
at com.ibm.team.repository.service.internal
Caused by:
com.ibm.db2.jcc.b.co: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=EPHSJ01, DRIVER=3.53.70
at com.ibm.db2.jcc.b.bd.a(bd.java:676)
at com.ibm.db2.jcc.b.bd.a(bd.java:60)
at com.ibm.db2.jcc.b.bd.a(bd.java:127)
at com.ibm.db2.jcc.b.fm.b(fm.java:21
2011-10-12 05:02:17,833 ERROR ository.service.internal.RdbRepositoryDataMediator - Failing ast query
-----------
WorkItemQueryModel r = WorkItemQueryModel.ROOT;
AstQuery q = (AstQuery) IDataQuery.FACTORY.newInstance(r);
q.select(r.itemId());
q.select(r.stateId());