It's all about the answers!

Ask a question

Issues with connector performance benchmarking .

Vivek Gupta (5195) | asked Sep 09 '11, 7:22 a.m.
We are trying to benchmark the performance of RTC-CMVC connector and do the necessary optimizations . After addressing all the know issues we are unable to figure out the reason for one particular case .

We use interop service to import and synchronize the work item . To increase the throughput we call "IncomingSynchornizeAndWait" through an executor service with 20threads .

Through java profiling we have observed that import goes on very fast initially and but towards the end (last 20-30 defects - irrespective of total defect count) , one or two threads goes into blocked or deadlocked state and the thread stack trace shows two methods most of the time :
a) findProxyByUri
b) findProxyByExternalState

During this time , the interop DEBUG logs also halts completely (no output at all) .

This deadlock gets resolved after some time automatically .

We tried to find a way to unblock this situaiton by :
- Changing async task values for total and interop in admin UI ,
- Monitored the CPU and Disk usage to ensure that RTC server CPU is not in 'wait" state for I/O (Moved Db2 to another server)
- Tuned the AIX servers for no limits on files and memory.
- Turned off logging
- Turned off indexing by suspending the indexer
- Turned on Logging at DEBUG level for ccm, jts , interop for all items in the .

but could not figure out anything in particular and thus posting here.

During the deadlock CPU remain busy doing something . This is noticed even after we create lot of work items through java api itself .

We suspect that some RTC processes gets kicked in towards last leg of import leading to blocking calls on "findItem" method and once RTC finishes its own work , the import thread unblocks to finish the remaining import .

Is there a way to curcumvent this issue since the impact for us is more than 60% in import time and this seems to increase with increase in number of defects for import.

So to summarize :
1) What is RTC doing after import has started or when a bulk workitem creation has finished
2) What is the way to delay that process so that import can finish asap and RTC can continue with its work later on ?

4 answers

permanent link
Vivek Gupta (5195) | answered Sep 13 '11, 12:58 a.m.

Team - Any pointers on this ?

permanent link
John Vasta (2.6k15) | answered Sep 13 '11, 1:06 p.m.
It's hard to say what might be going on; I'm not familiar with what might be happening in RTC when lots of work items are created quickly. I suggest the standard approaches to debugging Java performance and threading issues. The first thing I would look at is a Java dump when you think things are hung up or slow, to try and see what all the threads are doing.

permanent link
Vivek Gupta (5195) | answered Sep 13 '11, 2:00 p.m.
Hi John - We thought of profiling RTC server as well , but stopped on this forum for quick help at its not the RTC server which gets blocked but one of the client threads (mainly in findProxyByUri , findProxyByExternalState ) .
Is there anyone else i can chat with on ST for some quick pointers from server side ?

permanent link
John Vasta (2.6k15) | answered Sep 15 '11, 10:27 a.m.
The "find" methods in the client just issue queries to the RTC server, so it sounds like the server gets into a state where it is not responding to those queries very readily. That's why it would be interesting to see thread dumps from the RTC server when it is in that state.

Your answer

Register or to post your answer.