It's all about the answers!

Ask a question

Is there a limit to the maxThreads value for Tomcat server?


1
1
KrishnaKanth Naik (76511015) | asked Oct 18 '12, 3:50 a.m.
JAZZ DEVELOPER
When we see the error:
SEVERE: All threads (150) are currently busy, waiting. Increase maxThreads (150) or check the servlet status

The normal intuition is to increase the maxThreads. Using the Server Tuning guide , if we know the total number of contiguous users we can use the following formula to increase this limit as:
(max_clients * 50) + minSpareThreads

However, if we have a 100 contiguous users using SCM features, this number would shootup to 5025. Looking atGuidelines for maxThreads in a wiki site, its mentioned that if we are increase the number of MaxThreads to
 greater than 750, its better to use the clustered mode of Tomcat.

Could anyone please suggest if there what is the maxThreads value you would use for a setup with 1000 users, of which over 500 users are SCM users.

Env details:
RTC : 3.0
Tomcat Version: 5.5.30
OS: Windows 2008 R2 (64 Bit)
RAM: 24 GB
MaxHeap: default (4GB)
MaxUsers: 1600
MaxPeak users (license use): 1000

PS: These recommendations are for a server instance, on a machine that is hosting 10 other tomcat instances of RTC 3.0.

2 answers



permanent link
Paul Ellis (1.3k613) | answered Sep 20 '13, 8:31 a.m.
JAZZ DEVELOPER
edited Dec 02 '13, 6:01 a.m.

I do not think that there is a simple answer to this question, which I know can be a contentious and frustrating answer.

As you mention, the theory here was interpretted as, that you take the Tuning Guide for 3.x and you'd come out with a figure of 5025 based on 100 concurrent users.  I think the word concurrent is particularly challenging when sizing a CLM environment, as it really does depend on usage patterns, as well as the data and build makeup at each site.  Article Tuning the Rational Team Concert 4.0 server clarifies this further with:

For all the databases we currently support, the database automatically manages the number of concurrent connections or clients that can be served.  However, it is helpful to explicitly decide how many concurrent client requests you want to allow to process at a time. Lets give this value a name (max_clients ). Note that max_clients is not simply the number of users of the server. It is in fact much smaller than that because not all users are executing a request at the same time. If this is a new server that doesn't have historical data about what the maximum number of concurrent requests is, a good starting estimate is 10% of the number of clients expected to be using the server at any one time.

Of course, you also need to take a step back and ensure that you're compliant with Collaborative Lifecycle Management 2011 Sizing Guide and that you're within in the maximum values for a JTS/CCM (which you're already at the alert levels of with 500 SCM users) specified within CLM limitations, in order to ensure that your installation is not going to trip over once you make the necessary changes to the Tomcat thread pool.

To answer your question though.  I do not think that you can do much more than iteratively increase this setting as needed.  The largest problem I am seeing in the last year (admittedly, as my focus is deployment) is that too many sites are not monitoring their systems.  My colleague Jorge and I wrote an article on the deployment wiki part of jazz.net to ensure that if you had nothing else available to you, you could avail of JConsole (look in your RTC client bin area) to see how your JVM and webserver are behaving.

I am not just shamelessly plugging my article here, but from using this we have been able to see how many threads are actually being used.  The problem, as alluded to by the 3.0 tuning article, is the way in which Tomcat uses its pool.  In order to achieve a MaxClients*10 setting, you'd need something in the region of *50.  I am not sure how true this is with the CLM/RTC application, as correctly determining how many of your logged in users are concurrent is not simple.  I have seen a site similar to yours, consistently using around 350 tomcat threads.  Their initial setting was 11500.  Clearly far too high, but even with monitoring we settled on 750 threads to ensure that peak periods do not topple the server.

I have been speaking to some performance experts in-house. They have seen 500+ for this setting used quite often prior to Tomcat 7, but values this high on Websphere or Tomcat 7 wouldn't be advisable.

There are some other considerations here.  There are many asynchronous tasks running by the various applications on the Jazz platform, all of which require their own threads.  Indeed, the more registered users there are on the JTS, the more work the synchronizers in particular have to perform.

So, in conclusion.  I would initially setup a system where I expect 1000 peak users (an unlikely concurrency with 1600 users max; this is usually more like 10% - as a logged in user is not a concurrent user) to have an initial MaxThreads of 200; especially due to the other 10 tomcats running.

I would then monitor using JMX, again if nothing else is available, to see how this copes during the peak periods.  As more users are added and the closer I get to this limit, I would then increment the number of threads in small increments (50-100). 

It must be noted that our threads are purposefully designed to stay active for long periods of time to facilitate faster start-up times etc.  This often confuses WAS admins who see in their logs that there are hung threads, when really they are not.

In your situation where you are seeing the message, I would increment by no more than 100 and monitor.

I hope this helps!  I will continue to edit this post as more information is gathered and may convert it to a Wiki article.  I would also like anyone reading this to comment as to whether it helps; whether more information is required to fulfil their desired knowledge need when clicking on this post, etc...


permanent link
Piotr Aniola (3.7k11738) | answered Dec 02 '13, 5:29 a.m.
Please see the following workitem for further discussion of this topic https://jazz.net/jazz/web/projects/Rational%20Team%20Concert#action=com.ibm.team.workitem.viewWorkItem&id=170396



Comments
Paul Ellis commented Dec 02 '13, 6:02 a.m. | edited Dec 02 '13, 6:02 a.m.
JAZZ DEVELOPER

Thanks Piotr - I don't know how I missed that 4.0 article's update.  It slots nicely into explaining the initial rule of thumb, so I edited my answer appropriately.

Your answer


Register or to post your answer.


Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.