Performance when suspending change-sets
Hi,
I have a question concerning the suspending of change-sets. I use Rational Team Concert Beta 2, Version 0.6.0.I20080311-1541 and its corresponding server. My scenario is the following: I have about 80 changed files, so 80 pending changes, all associated to the same work item. Now I stop working on the WorkItem and want the change-sets suspended. This operation lasts about 15 Minutes with the server machine (Core2Duo, 1,86GHz) running at 100% cpu usage (1 core). The server is started with -Xmx700m, according to the taskmanager about 300m are used. There are no other users connected to the jazz server. For me as user, this seems a very simple operation, so I do not understand what causes the enormous long time of 100% CPU usage? Can anyone shed some light on this issue? Thanks, P.Schaich |
10 answers
Suspend should be quick, it does have to update the files that are suspended, but the server should never be pegged by this operation.
There is a video at: http://jazz.net/pub/learn/videos/data/jazz-scm-test-drive.swf which shows at 3:10 suspending of 100 changes. It's fast. The setup for the video was Derby with Jetty and the M5 client. Could you tell me more about your setup? If you are running with Tomcat you should be able to get a java stack dump to see what is running at the time the CPU is pegged. Jean-Michel Lemieux Jazz Source Control Team |
|
Hi Jean-Michel,
thanks for the fast response. I will check the possibility for a java stack dump in the next days and come back to this. The jazz server is set up as is (Tomcat and Derby). The jazz scm was setup by doing a cvs import (~15000 source files, 3 years cvs history) and bugzilla import (~1000 workitems). The backuped derby database is around 1 GB in size. So the scm itself is kind of bigger than a usual workshop scenario, I suppose. Kind Regards, Peter Edit: So, here is the stack dump: http://thefilehost.net/download.php?id=556F7C891 |
|
Hi Peter,
That stack trace shows that the server is idle, all threads are waiting for connections. One idea is that you've pushed the boundaries of Derby, but we are going to be testing this ourselves a bit more this week. It doesn't seem like your data-set is that big, so we should be able to handle this with Derby. We've seen commercial DBs have much better performance, but with smallish data-sets and # of users, it should do fine. If you can get a trace when the server is at 100% CPU that would be great, and I'll get back with the DB size we've tested this week on Derby. Jean-Michel Lemieux Jazz Source Control Team |
|
Hm, this is strange: I hit Ctrl-Break while it was in 100% cpu usage, but sure, I will try again.
Edit: so here are some additional stack dumps, and also a screenshot of the taskmanager: http://www.thefilehost.net/download.php?id=A70EDEBB1 The good news is to hear from you that this size of repository should hopefully manageable by the jazz scm, tomcat and derby. Some additional info: After observing this 100% cpu usage problem a little bit, it occurs in almost any operation (deliver, accept changes, revert changes, etc), but not this long time as in my original described scenario. The Windows XP is run inside VMWare 6.0.3, and is currently the only image running on the host system. |
|
So, some updates:
I investigated a little bit how to tune up the server myself, and upgraded the XP in the VMWare image to have more memory (1.5 GByte) and added the second core. Additionally, I increased the java memory with -Xmx1100 and gave derby some more pageCache by -Dderby.storage.pageCacheSize=40000. Now, the suspending of the change-sets lasts around 5 minutes, but still gives plenty cpu usage to the server. Here are some stackdumps again: http://www.thefilehost.net/download.php?id=D2409A461 So, nevertheless we can work so far, but are sure interested in assistance to improve the performance, or at least know the factors that influence the performance behaviour to avoid them in future. |
|
Thanks for the stack traces. This really looks like the entire server is running slowly, mostly IO bound.
Yesterday I setup a Derby DB with the linux kernel (23K files) as source and 6 years of CVS history of around 4000K files. The Derby + Jazz setup was very performant. The only thing that seems different in our setups is VMware. Can you try running natively? |
|
I did a quick check of the disk performance (since you mentioned IO bound), and yes, inside the VMWare image,
the disk performance is kind of slowish compared to the host system (~4 times slower) I will investigate further, but this is for sure something that will put breaks on the Jazz server performance. Thanks very much so far, I will come back to it when I get time to test it outside the vmware image. |
|
So, I had time now to increase IO performance on the VMWare image and additionally to test it on real hardware, and unfortunately, I doesn't change much.
CPU usage remains as high as reported, and IO performance to the harddisk seems not to be as important as guessed. The only difference I currently see is, that I tested it using Windows XP, and you tested it using Linux, but this should - in my opinion- not concern the cpu usage. Unfortunately, I don't have the possibility to test it on a linux host. Jean-Michel , do you have any suggestions how I can assist to improve this for future releases? |
|
I think the best thing now is to open a defect and provide as much information as possible about your setup and data-set sizes and we will investigate further. Be sure to include:
1. how much data did you import into the jazz repo? how many files, did you import any history? 2. platforms for both server and client 3. bonus marks if you can install YourKit profiler (there is a free evaluation license on yourkit.com) and profile the server while the suspend is running and attach the profile dump. Jean-Michel |
We have fixed some performance regressions with Derby in RC1, when it is
available I recommend trying it out. I still (occasionally) see CPU spikes with embedded Derby that I am uncomfortable with but we are hoping to find a way around this. Thanks, JohnC SCM Server schaich wrote: So, I had time now to increase IO performance on the VMWare image and |
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.