What does com.ibm.team.repository.service.internal.RepositoryRemoteService.fetchOrRefreshItems do?
Any idea what this activity would accomplish and how it could bring RTC to its knees?
Accepted answer
JVMDUMP032I JVM requested Snap dump using 'D:\IBM\rtc_502\
JVMDUMP010I Snap dump written to {nothing to snap}
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP032I JVM requested Java dump using 'D:\IBM\rtc_502\
JVMDUMP032I JVM requested Heap dump using 'D:\IBM\rtc_502\
This active service ran out of memory every time and hung RTC.
We had an emergency temp memory upgrade to the max 32GB allowed by Window server 2008, but the allocation according to the rule of thumb 50/50 heap/OS of -Xmx16G/32GBmem still did not allow completion of this job.
We finally broke out of severity 1 by using a skew allocation of higher heap, with the job completed at 4hr 26 min
We were not aware of the indexing catch up job(s) which would be long running after our routine huge imports of WI's, though this time confirmed by project team after the fact that it was bigger than our huge norm. Always thought before that indexing would complete along with each successful import.
One other answer
Dependent on your topology and the applications you have actually installed candidates for high loads are the Data Collection Component, BIRT reports and potentially others. It would be a good idea to also look at the database and search for errors there as well, as if the DB starts failing the application might show a bad performance as well.
Just the CPU load does not really help either that is why you should have set up server monitoring to see how the server performance changes over time.
It is unlikely to detect the root cause without a lot more communication and information which support is able to get and share but the forums are likely not.
Comments
Thx Ralph,