Improving the performance of Rational Team Concert by using the dynamic cache service in WebSphere Application Server

For WebSphere DynaCache configuration with Rational Team Concert version 6.0.1 or higher, see Configuring WebSphere DynaCache for Rational Team Concert server

IBM Rational Team Concert has a web client that is useful for managing work items without the development Eclipse client. When you use the web client in a production environment, from a performance point of view, it is important to consider how many users are concurrently accessing the system. Too many concurrent users can cause bottlenecks in the server layer for Rational Team Concert and overall for the Change and Configuration Management (CCM) system.

When you use the work item capability of Rational Team Concert in the web client, the system generates several internal HTTP requests to the application server. If the process configuration uses many work item types and custom attributes, the possibility of performance problems increases. Because the IBM WebSphere Application Server dynamic cache (DynaCache) service enables some of the HTTP requests responses to be cached by the server, it is important to analyze the behavior of the system and use the available caching techniques to improve performance and enable more concurrent users without increasing the hardware infrastructure. Because Rational Team Concert can be deployed on top of WebSphere Application Server, you can use the WebSphere Application Server DynaCache service to cache the responses to the HTTP requests and improve the performance and behavior of the system.

With the DynaCache service, if the application server receives an HTTP request that is the same as a previous request and the system state has not changed for the given parameters, the requested service is not run. Instead, the previously cached response is sent back to the browser. This behavior improves performance and response time and reduces the load on the web server for other requests. The use of the DynaCache service reduces performance and contention problems, even when working with many work items, custom attributes, or concurrent users.

To better understand WebSphere DynaCache, see the following IBM Redbooks publication:

Mastering DynaCache in WebSphere Commerce (SG24-7393-00)

Although the publication is written for WebSphere Commerce, the behavior also applies to Rational Team Concert because both solutions are deployed on WebSphere Application Server.

Activating the DynaCache service

To activate the DynaCache service by using WebSphere Application Server, you must access the application server console, which is typically in one of these locations:

https://yourserver:9043/ibm/console
https://yourserver:9060/ibm/console

where yourserver is the host name or IP address of the server, and 9043 or 9060 is the port. Depending on your configuration, you might need to specify other ports.

You also must have the right permissions to log in and use the administrative console where Rational Team Concert is deployed.

To enable the DynaCache service, open the WebSphere administrative console and go the Web container configuration page. Select the Enable servlet and command caching check box:

Figure 1. Enabling the DynaCache service in WebSphere Application Server

Important: After you select the Enable servlet and command caching check box, you must restart the web server. When that check box is selected, the WebSphere DynaCache service is enabled. Depending on your configuration, you may also need to uncheck the Enable Edge Side Include check box for IBM HTTP Server. For instructions, see the next section, "Configuring IBM HTTP Server".

Configuring IBM HTTP Server

During the activation of the DynaCache service, you might also need to configure the IBM HTTP Server. To avoid compatibility problems, open the IBM HTTP Server console and uncheck the Enable Edge Side Include check box:

Figure 2. Configuring the IBM HTTP Server

Configuring the DynaCache service

After the DynaCache service is enabled, edit the cachespec.xml file to configure which requests are going to be cached for the .war files that are used in your application.

1. Go to the folder where the Rational Team Concert .war files are deployed.
2. Open the .war file to configure.
3. Find the WEB-INF/cachespec.xml file.
4. Edit the file to configure caching.

During the deployment that this article uses an example, performance issues were observed in the services that the ccm.war file provided. This article focuses on the ccm.war file.

For this example, on a Windows server, go to the folder where WebSphere is installed and find the location of the CCM cachespec.xml file. In this example, for the ccm_war, the path is WebSphere/AppServer/profile/AppSrv01/installedApps/serverNode01Cell/ccm_war.ear/ ccm.war/WEB-INF.

Figure 3. Locating the cachespec.xml file on a Windows system

This XML file contains all of the directives to cache the requests that are received by CCM. The syntax of this XML is described in more detail in Section 2.6 of the DynaCache Redbooks publication. The following figure, Figure 2-10 from the Redbooks publication, and shows the general structure of the cachespec.xml file.

Figure 4. The cache-entry format from figure 2-10 in the DynaCache Redbooks publication

Figure 2-10 shows the structure of the cachespec.xml file. Each cache-entry has a class that defines the behavior of the caching, as well as parameters to configure the cache-entry. In this example, the "static" class is used to define static elements, and "servlet" is used to define the caching of REST and other service responses.

The following figure, Figure 2-12 from the same Redbooks publication, shows an example of how the DynaCache service can be configured to cache specific HTTP requests.

Figure 5. Configuring DynaCache, from figure 2-12 in the DynaCache Redbooks publication

The example in Figure 2-12 shows how caching works for a shopping cart page for a specific item in a store.

Configuring the cache to handle static content is easier than configuring the cache to handle dynamic content. The configuration for static content is like the following example, which caches responses for a request to a static resource:

<cache-entry>
  <class>static</class>
    <name>/_theming/resource/images/banner-sprites.png</name>
    <cache-id>
      <inactivity>288000</inactivity>
    </cache-id>
</cache-entry>

When the class element content is "static", the configuration declares an element that does not change. In this case, the resource named "/_theming/resource/images/banner-sprites.png" is declared to be static, so it will not change. Static elements will be served from the cache until the inactivity time reaches the second value specified in the inactivity tag. In this case, after 288000 seconds (8 hours) of inactivity, the image will be removed from the cache. The default value is 0, which means it never expires.

When the class is "servlet", the entry might look like this example:

<cache-entry>
   <class>servlet</class>
   <name>/service/com.ibm.team.workitem.common.internal.rest.IWorkItemRestService/editorPresentation</name>
   <cache-id>
      <inactivity>288000</inactivity>
      <component id="typeId" type="parameter" />
      <component id="itemId" type="parameter" />
      <component id="projectAreaItemId" type="parameter" />
  </cache-id>
</cache-entry>

In this case, the response from the REST service is cached and served when the typeId, itemId and projectAreaItemId are the same. Caching REST service responses can greatly increase performance. However, the cache must currently be manually cleared when the response is changed and this happens when the process template is changed. In this example, the cache would need to be manually cleared when the Rational Team Concert Project Area configuration changes the presentation (page layout) for a particular work item type. In essence, if the process template is changed, it is recommended to clear the content cache using the CacheMonitor application to avoid problems caching old contents. The steps for clearing the cache are described below in the the "Monitoring and clearing the DynaCache service" section.

As part of this example, elements were cached in a third way by creating a proxy to the existing REST services and hard coding the cache behavior. For more details, see the next sections.

Improving performance in Rational Team Concert by using the DynaCache service

When the Rational Team Concert repository server is deployed by using WebSphere Application Server, you can enable the DynaCache service to cache dynamic content and improve performance. You configure the cache to handle static, servlet, and REST service elements and insert a proxy in the REST services to add caching behavior.

Configuring DynaCache behavior in the cachespec.xml file

The DynaCache service requires that each request be identified by a unique name which is a part of the URI, for example _/service/com.ibm.team.workitem.common.internal.rest.IWorkItemRestService/linkTypesForProject _. The cachespec.xml file uses the name as well as the parameters, cookies, or other request characteristics to devise a key for the cached element. For the Rational Team Concert system the cache directives are based on either the input parameters of the request or cookies to create a unique key in the DynaCache map.

The cache-entry in the earlier section shows the entry for the editorPresentation service. Over 40 cache-entry values exist. Here are a few more examples:

<cache-entry>
   <class>servlet</class>
   <name>/service/com.ibm.team.workitem.common.internal.rest.IWorkItemRestService/linkTypesForProject</name>
   <cache-id>
      <inactivity>288000</inactivity>
      <component id="projectAreaItemId" type="parameter" />
   </cache-id>
</cache-entry>

<cache-entry>
   <class>servlet</class>
   <name>/service/com.ibm.san.wiext.common.service.IContributorRolesRestService/contributorRolesIds</name>
   <cache-id>
      <inactivity>288000</inactivity>
      <component id="projectAreaId" type="parameter" />
      <component id="teamAreaId" type="parameter" />
      <component id="userId" type="parameter" />
   </cache-id>
</cache-entry>

<cache-entry>
   <class>static</class>
   <name>/web/dojo/number.js</name>
   <cache-id>
      <inactivity>288000</inactivity>
   </cache-id>
</cache-entry>

<cache-entry>
   <class>static</class>
   <name>/web/com.ibm.team.workitem.web/ui/internal/view/common/images/pixel.gif</name>
   <cache-id>
      <inactivity>288000</inactivity>
   </cache-id>
</cache-entry>

The attached cachespec.xml file includes the specifications for all of the elements that were cached as part of this example.

Optimizing the cache behavior in the REST service

Note: Currently, Rational Team Concert does not support the use of plug-ins to introduce proxy services. In the environment in this example, the improvement was significant enough to accept the risk of using an unsupported capability.

Two specific requests caused the system to spend a lot of time generating a result when many users or many work items existed. These REST services were as follows:

• workItemDTO2: This HTTP request returns the value of each field in the work item
• workItemEditableProperties: This HTTP request returns the required and read-only fields

To handle these requests, proxies were created in Java servlet code to update the REST services to cache the responses and to control the behavior of the cache. The proxies were necessary because the cache behavior could not be configured in the cachespec.xml file; not all of the values that were needed to build the unique key could be derived from the URL and request parameters. The proxy also allows you to invalidate the cache if the requested work item changed since the cache entry was created. The work item hasWorkItemChanged REST service is used to determine whether the work item changed since the last state was obtained.

These service proxies are installed as plug-ins and extend the WorkItemRestService behavior to check and manage the cache. The plug-ins override the REST service methods and checks and maintains the responses in the cache for the workItemDTO2 and workItemEditableProperties REST services. With the proxy installed, the actual services are implemented in the superclass.

The caching of the workItemEditableProperties HTTP request is managed according to the role of the user, state/transition in the workflow, and type of the work item. For example, consider two users: Anita and Tracy, both of whom have the same role. In this case, if they each make a request for the same work item with the same parameters, the stored data can be cached, and a request by Anita will improve the performance of a subsequent request by Tracy. This is because with the same role, all the read-only and required attributes would resolve to be the same. If, however, Anita has a different role, then they must each have their own cached entry, since the role can determine the values and behavior of certain attributes and presentations.

When a work item is loaded, the workItemDTO2 service is called; then, the workItemEditableProperties service is called. In this example, the environment has many work item types and custom attributes, so the combined time to invoke these REST services was about five seconds. By caching the workItemDTO2 REST service, response time was improved for these services to a half second. The improvement for the workItemEditableProperties REST service was also significant.

The files that are attached to this article include the configuration files and plug-ins that were used to enable caching.

If you look at the Java code, you can see that you first must look up the DynaCache that is defined in the WebSphere configuration:

/*
 * WebSphere DynaCache initialization
 */
private static Map<String, Object> workItemDynaCache = null;

Context initialContext = jndiService.getInitialContext();

workItemDynaCache = (Map<String, Object>)initialContext.lookup("services/cache/basecache");

The DynaCache initialization code uses the Java JNDI interface to communicate with the cache. The JNDI name, "services/cache/basecache", identifies the specific cache. By using the default cache name in the look-up, you use the same cache that is activated in the administration console, so the REST HTTP requests also are shown in the console. You can use the same cache monitor application to control the content.

Next, you must build a unique key for each entry in the DynaCache. This example focuses on two REST services, workItemDTO2 and workItemEditableProperties. For each service, the key was built in a different way to ensure that they keys were unique.

/*
 * Generate key for workItemDTO REST service
 */
private static final String WORKITEM_DTO_CACHE_KEY = "workItemDTO2?includeHistory=%b&id=%d";

private String getKey(int id, boolean includeHistory) {
    return String.format(WORKITEM_DTO_CACHE_KEY, includeHistory, id);
}

private static final String WORKITEM_EDITABLE_PROPERTIES_CACHE_KEY = "workItemEditableProperties?projectAreaItemId=%s&roleId=%s&workItemType=%s&stateId=%s&workflowActionId=%s"; 

/*
 * Generate key for workItemEditableProperties REST service
 */
private String getKey(String projectAreaItemId, String roleId, String workItemType,
                                           String stateId, String workflowActionId) {
    return String.format(WORKITEM_EDITABLE_PROPERTIES_CACHE_KEY, 
                    projectAreaItemId, roleId, workItemType, stateId, workflowActionId);
}

In the custom code shown below -- and included in attachment com.ibm.team.workitem.service.zip -- a unique key is created based on the project area, role, type, state, and action. In the case of workItemEditableProperties, the key is calculated as follows:

/*
 * Generate Key based on itemId
 */
public String getKey(ParmsGetAllValues parms) throws TeamRepositoryException {
    try {
   // Get the work item
   IWorkItem workItem = getWorkItem(parms.itemId, null);

   // Find the current Team Area that work item belongs to
   IProcessArea processArea = findProcessArea(workItem);

        // Find the role, state and work item type
   String roleId = getMainRole(service.getAuthenticatedContributor(), processArea);
   String stateId = workItem.getState2().getStringIdentifier();
   String workItemType = workItem.getWorkItemType();

        // Generate a key from the ids
   String result = getKey(parms.projectAreaItemId, roleId, workItemType, stateId, parms.workflowActionId);
   if (LOGGER.isDebugEnabled()) {
      LOGGER.debug(String.format("WorkItemEditableProperties <- workItemId = %i", workItem.getId()));
        LOGGER.debug(String.format("WorkItemEditableProperties -> key = %s", result)); 
    }

    return result;
    } catch (IllegalArgumentException iae) {
    return null;
    }
}

The DynaCache object is a Map collection with a key and a value. The value can be an object. You can add and remove elements. Make sure that you are able to do these tasks:

• Build a unique key
• Detect when the value must be invalidated and removed from the cache. For example, workItemDTO2 is invalidated when hasWorkItemChanged REST service returns true.

By using the Java cache manager, you can save many seconds by running these requests. In this example, for workItemDTO2 requests, the response time was decreased from 4.7 seconds to 472 milliseconds:

Figure 6. REST service timing when loading a work item without caching

Figure 7. REST service timing when loading a work item with caching

Performance before and after using the DynaCache service

To compare the performance before and after the DynaCache service was used, run the web client on Firefox, and make sure that Firebug is configured to show HTTP activity. When a work item editor is loaded, various requests are made to load the work item and its presentation properties. In this example, 30 HTTP requests are made to the application server when the work item is loaded:

Figure 8. Page load timing when loading a work item without caching

If you open the same work item a second time in the same tab in the same browser, you see only one HTTP request:

Figure 9. Page load timing when you load a work item for the second time in the same browser tab without caching; only one service call is used

However, if you open the same work item in another tab or browser or if another user tries to open the same work item, the same 30 HTTP requests as those that were made the first time are shown.

Figure 10. Page load timing when you load a work item in a second browser tab without caching on a 32 processor system

The same requests are also sent when a second user opens the same work item. By using the DynaCache service, you can reuse the responses of many of the HTTP requests to decrease the response time and increase the number of concurrent users that can use the system with good performance.

The responses of HTTP requests were reused in a real customer environment that had many work item types and many custom attributes. Before the customer used the DynaCache service, the customer was experiencing problems with the performance and the scalability of the system. The system could not be used in production because the CPU utilization on the application server side was continually at 100%, so users were locked out and could not work. Although the number of CPU cores was increased from six to thirty-two, the problem continued. For this customer's configuration, the system did not scale.

To solve this problem, the DynaCache service was used. The scalability of the system increased and the response time improved dramatically. The system behavior was improved so much that with the DynaCache service, the number of processors could be decreased back to the six original CPUs without problems. In addition, the response time and scalability of the system improved.

In another example, when a highly customized work item was opened in the production environment, 78 HTTP requests were made to the application server for one user to see one work item. Many of these requests were concurrent. Of these requests, a few specific service calls spent the most time. These requests consumed the CPU cores of the application server without allowing the server to be used for other users. These issues caused poor response time of the system, and the server needed to be shut down in production several times.

In this example, the first time a work item is opened, 78 requests are made:

Figure 11. Page load timing when loading a work item without caching

The area near the top with many bars shows how the request taxed the server. As before, if the same work item is opened in a different tab or a different browser or is requested by a different user, the same 78 HTTP requests are run, wasting the resources of the server.

By enabling the DynaCache service and configuring the cachespec.xml in CCM.war as shown below, the response time for each cached HTTP request was greatly improved. In turn, the response time for the entire the server was improved, so more requests from more users could be processed at the same time. The following screen capture shows the HTTP requests to open the same work item that was included before, but with the DynaCache service enabled and configured:

Figure 12. Page load timing when loading a work item with caching

The screen shows several requests serviced from the cache, and the reduced load on the server.

Note: When a request is stored in the DynaCache service into WebSphere Application Server, you can see an extra header in the response:

Figure 13. The DynaCache service adds an extra heading parameter when a cached response is returned

Compare the system load before and after the DynaCache service was used. Before, the system was heavily loaded even with 32 processors. The following graph shows the high CPU utilization spikes:

Figure 14. CPU utilization with 32 processors before using the DynaCache service

After, the behavior of the server improved.

Figure 15. CPU utilization with 32 processors after using the DynaCache service

After using the DynaCache service with 32 processors, the CPU usage went from 100% to less than 10% in the peak times. The following graph shows the production CPU utilization with only six processors.

Figure 16. CPU utilization with six processors after using the DynaCache service

The percentage of the usage of the CPU is higher than the earlier graph because there are fewer processors.

In the customer environment, after the first day in production, the cache monitor information showed that 123,423 requests to the application server were returned by using the information that was stored in the cache. That is a high number of requests that were not run in the application server, freeing up capacity to run other new requests. The following screen capture shows the monitor result (in Spanish):

Figure 17. First day cache hit count in the production system

As of this writing, the environment for this customer has been in production for several months without problems. The production team returned to using only six processors instead of the 32 that were needed before the DynaCache service was used. The behavior of the system is good, even with five times fewer processors.

Monitoring and clearing the DynaCache service

WebSphere Application Server includes a monitor that allows control of the status and contents of the cache in real time. To use it, you locate the CacheMonitor.ear file in the WAS_installdir/installableApps directory and then use the WebSphere configuration screens to install it as a common J2EE application.

The process to install the monitor is easy. The following screen captures show the WebSphere Application Server version 7 administrative console. Other versions of the application server are similar.

1. Open the Enterprise Applications page.
2. Click Install.
   
   
   Figure 18. Enterprise Applications
   
   
3. Browse to select the .ear file to install.
   
   
   Figure 19. Path to the new application
   
   
   Figure 20. Select the .ear file
   
   
4. Click OK to return to the path screen.
   
   
5. Click Next to open and specify the installation options.
   
   
   Figure 21. Path to the new application
   
   
6. Click Fast Path and then click Next to configure the additional installation options.
   
   
   Figure 22. Installation options
   
   
   Figure 23. Additional installation options
   
   
7. Click Next to map modules to servers.
   
   
   Figure 24. Map modules to servers
   
   
8. Click Next to review your selections.
9. Click Finish to complete the installation. The install log shows your installation results.
   
   
   Figure 25. Review installation selections
   
   
10. When the installation finishes, click Save directly to the Master configuration.
    
    
    Figure 26. Install log
    
    
11. Start the Dynamic Cache Monitor.
    
    
    Figure 27. Start the monitor from the main page
    
    
    Figure 27. The main page shows the status of the applications
    
    The Dynamic Cache service is now started.
    
    
12. After the application is installed and started, view and manage the cache information by using a URL that is in this format:

    https://yourserver:9043/cachemonitor

    where yourserver is the host name or IP address of your server, and 9043 or 9060 is the port. Depending on your configuration, other ports might need to be specified. For example, https://ipServer:portApplication/cachemonitor.
    
    The monitor console opens:
    
    
    
    Figure 28. The Cache Monitor shows the cache statistics
    
    

By using the monitor, you can view the cache statistics, cache contents, and the cache policies that are included in cachespec.xml files in real time. If you have the right permissions, you can also clear the cache without stopping the service.

In the monitor, you can explore information and clear the cache contents. This screen capture shows that the cache policy is currently configured to cache the /ccm/web/dojo/resources/blank.html resource, which is a static element. Any framework or application static element can be safely cached.

Figure 29. Cache instance policies

The Cache Contents selection shows the content of the cache (empty) and a button that you can click to clear the cache.

Figure 30. Cache instance content

For detailed information about the cache monitor, see IBM Knowledge Center at http://www-01.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.base.doc/ae/tdyn_servletmonitor.html.

Conclusion

The DynaCache service can be used to improve response time, increase stability, and reduce the processing power and energy used to run a solution. This approach was especially effective in a customer environment that contains many users work item types. Caching static and servlet elements by using the configuration in the cachespec.xml file provided a base level of performance improvement. In addition, for REST services that could not be configured in the cachespec.xml file, a plug-in was used to enable caching by installing proxy REST services that use JNDI to communicate with the cache.

Using the DynaCache service is an effective way to improve performance in IBM Rational Team Concert, or in any web server environment that is processor- or resource-constrained.

Resources:

• cachespec-ccm.xml: Sample cachexpec-ccm.xml file.
• com.ibm.team.workitem.service.zip: Extension to implement DynaCache in the REST service.

External links:

• WebSphere Cache Monitor Knowledge Center
• Mastering DynaCache in WebSphere Commerce (SG24-7393-00)