Issues with connector performance benchmarking .
We are trying to benchmark the performance of RTC-CMVC connector and do the necessary optimizations . After addressing all the know issues we are unable to figure out the reason for one particular case .
We use interop service to import and synchronize the work item . To increase the throughput we call "IncomingSynchornizeAndWait" through an executor service with 20threads .
Through java profiling we have observed that import goes on very fast initially and but towards the end (last 20-30 defects - irrespective of total defect count) , one or two threads goes into blocked or deadlocked state and the thread stack trace shows two methods most of the time :
a) findProxyByUri
b) findProxyByExternalState
During this time , the interop DEBUG logs also halts completely (no output at all) .
This deadlock gets resolved after some time automatically .
We tried to find a way to unblock this situaiton by :
- Changing async task values for total and interop in admin UI ,
- Monitored the CPU and Disk usage to ensure that RTC server CPU is not in 'wait" state for I/O (Moved Db2 to another server)
- Tuned the AIX servers for no limits on files and memory.
- Turned off logging
- Turned off indexing by suspending the indexer
- Turned on Logging at DEBUG level for ccm, jts , interop for all items in the log4.properties .
but could not figure out anything in particular and thus posting here.
During the deadlock CPU remain busy doing something . This is noticed even after we create lot of work items through java api itself .
We suspect that some RTC processes gets kicked in towards last leg of import leading to blocking calls on "findItem" method and once RTC finishes its own work , the import thread unblocks to finish the remaining import .
Is there a way to curcumvent this issue since the impact for us is more than 60% in import time and this seems to increase with increase in number of defects for import.
So to summarize :
1) What is RTC doing after import has started or when a bulk workitem creation has finished
2) What is the way to delay that process so that import can finish asap and RTC can continue with its work later on ?
We use interop service to import and synchronize the work item . To increase the throughput we call "IncomingSynchornizeAndWait" through an executor service with 20threads .
Through java profiling we have observed that import goes on very fast initially and but towards the end (last 20-30 defects - irrespective of total defect count) , one or two threads goes into blocked or deadlocked state and the thread stack trace shows two methods most of the time :
a) findProxyByUri
b) findProxyByExternalState
During this time , the interop DEBUG logs also halts completely (no output at all) .
This deadlock gets resolved after some time automatically .
We tried to find a way to unblock this situaiton by :
- Changing async task values for total and interop in admin UI ,
- Monitored the CPU and Disk usage to ensure that RTC server CPU is not in 'wait" state for I/O (Moved Db2 to another server)
- Tuned the AIX servers for no limits on files and memory.
- Turned off logging
- Turned off indexing by suspending the indexer
- Turned on Logging at DEBUG level for ccm, jts , interop for all items in the log4.properties .
but could not figure out anything in particular and thus posting here.
During the deadlock CPU remain busy doing something . This is noticed even after we create lot of work items through java api itself .
We suspect that some RTC processes gets kicked in towards last leg of import leading to blocking calls on "findItem" method and once RTC finishes its own work , the import thread unblocks to finish the remaining import .
Is there a way to curcumvent this issue since the impact for us is more than 60% in import time and this seems to increase with increase in number of defects for import.
So to summarize :
1) What is RTC doing after import has started or when a bulk workitem creation has finished
2) What is the way to delay that process so that import can finish asap and RTC can continue with its work later on ?
4 answers
It's hard to say what might be going on; I'm not familiar with what might be happening in RTC when lots of work items are created quickly. I suggest the standard approaches to debugging Java performance and threading issues. The first thing I would look at is a Java dump when you think things are hung up or slow, to try and see what all the threads are doing.
Hi John - We thought of profiling RTC server as well , but stopped on this forum for quick help at its not the RTC server which gets blocked but one of the client threads (mainly in findProxyByUri , findProxyByExternalState ) .
Is there anyone else i can chat with on ST for some quick pointers from server side ?
Is there anyone else i can chat with on ST for some quick pointers from server side ?