We recently installed a custom Jazz plugin developed by IBM teams in China, called EasyInspect(com.ibm.team.rtc.inspection.common_1.0.0.jar), to our Jazz server. It required us to restart Jazz server and reinstall licenses as all files are removed under /opt/IBM/WebSphere/AppServer/profiles/SERVERNAME/temp/NODENAME/server1/jazz_war/jazz_war.

We first did the test on our test instance server and it seems ok. So we installed on our production server. But the story is compeletly different. When trying to restart Jazz, it took more than 20 minutes and then another 20 mi nutes to get on to the Jazz admin web page to reinstall the licenses. After that it took me another 15 minutes to open a work item.

And then it gto worse as we started having failures, starting with this log line (in jazz.log.16):

2009-04-08 00:20:01,103 WARN sqlExceptionLogger &nb sp; &nb sp; &nb sp; - SQL: INSERT INTO COMMON_SNAPSHOT.TIME_TBL (TIME_ID, TIMESTAMP_COL, DAY_COL, WEEK, MONTH_COL, YEAR_COL) VALUES (?, ?, ?, ?, ?, ?)
SQL Exception #1
SQL Message: Exception occurred during Timestamp conversion. See attached Throwable for details.
SQL State: null
Error Code: -99999

After the few initial errors there were repeating errors that caused the log to scroll continuously:

2009-04-08 00:21:09,833 WARN com.ibm.team.repository &nbs p; & nbsp; &nbs p; & nbsp; &nbs p; - CRJAZ0989I The log feed change event appender failed to list the pending log events from the directory "/opt/IBM/WebSphere/AppServer/profiles/ AppSrv01/temp/ips-searchNode01/server1/jazz_war/jazz.war/eclipse/ configuration/org.eclipse.osgi/bundles/39/data/logFeedAppender".
2009- 04-08 00:21:10,819 WARN m.team.repository.service.internal. LogFeedAppender - CRJAZ0994I Failed to write log event to disk.

Trying to attach to websphere console and looking at processor usage showed that WAS was *very slow*. I tried to stop it cleanly using stopserver.sh but gave up after 15 minutes. Kill -9 and restart of the server has made the problem go away for now.

Please help us invetigate to find cause and ways to prevent it from comeing back again.