There are specific user and system scenarios across the CLM portfolio that can potentially drive higher load on a Jazz application. Starting with CLM 6.0.4 IBM has identified a set of known resource-intensive scenarios in each application and are exposing them as JMX MBeans. Refer
Known Resource-intensive Scenarios
However there are cases where a custom functionality could be a potential resource-intensive scenario, for example a custom Java script or a plugin making REST APIs call to a CLM application. Such custom functionality needs to be monitored as well when looking at the overall application performance. This article focuses on a method to record and publish such custom functionality as resource-intensive scenario to help track their usage.
About Starting and Stopping Custom Resource-Intensive Scenarios
The workflow is to first register a JMX MBean stating that a Resource Intensive Scenario is about to start before any Script/API first accesses the targeted application Server. At the end of the script , once the operation is complete, update the MBean stating the specific Resource Intensive Scenario has completed. This MBean has several attributes like Start, End, isActive, Username, Elapsed time and invocation Count. These MBean attributes can be monitored/tracked and relevant notifications/thresholds can be configured against them.
Let us first look at the workflow of registering a Resource Intensive Scenario MBean using a REST client including monitoring via RepoDebug. We will then present different examples using APIs and Scripts
Workflow of Registering a Resource-Intensive Scenario using REST Client
To perform any of the activities below, you must be logged in to the application
https://<servername>:<port>/<APP>/web
Starting a Scenario
To start a Scenario MBean make a POST request to the URL
https://<servername>:<port>/<APP>/service/com.ibm.team.repository.service.serviceability.IScenarioRestService/scenarios/startscenario
Provide the following headers:
Content-Type=application/json
Accept=application/json
X-Jazz-CSRF-Prevent=JSESSIONID
Where
JSESSIONID
is the session ID value of the logged-in session. If this header is not provided the call will not succeed.
The POST body must contain the scenario name in the following JSON format.
{"scenarioName":"<your scenario name here>"}
Example:
{"scenarioName":"MyCustomExpensiveScenario"}
If the call succeeds, the response returns 200 as HTTP status. The response body contains the scenario registration information required to register the end of the scenario later. In the example the scenario name used was MyCustomExpensiveScenario.
{
"scenarioName":"MyCustomExpensiveScenario",
"scenarioInstanceId":"_Jbe94DaQEempFf7xSdsBAQ",
"scenarioHeaderKey":"x-com-ibm-team-scenario",
"scenarioHeaderValue":"_Jbe94DaQEempFf7xSdsBAQ%3Bname%3DMyCustomExpensiveScenario"
}
Stopping the Scenario
To stop a Scenario MBean once the scenario has ended, make a POST request as below
https://<servername>:<port>/<APP>/service/com.ibm.team.repository.service.serviceability.IScenarioRestService/scenarios/stopscenario
Provide the following headers:
Content-Type=application/json
Accept=application/json
X-Jazz-CSRF-Prevent=JSESSIONID
Where
JSESSIONID
is the session ID value of the logged-in session. If this header is not provided the call will not succeed.
The POST body must contain the scenario information from the start response in JSON format. Example:
{"scenarioName":"MyCustomExpensiveScenario",
"scenarioInstanceId":"_Jbe94DaQEempFf7xSdsBAQ",
"scenarioHeaderKey":"x-com-ibm-team-scenario",
"scenarioHeaderValue":"_Jbe94DaQEempFf7xSdsBAQ%3Bname%3DMyCustomExpensiveScenario"}
If the call succeeds, the response returns 200 as HTTP status. The response body contains the scenario ID. Example:
"_Jbe94DaQEempFf7xSdsBAQ"
Monitoring or Viewing the Scenario MBean
You must first enable specific MBeans so as to be able to Monitor it. To enable the MBeans, Login to the application admin page as a Jazz Admin User via
https://<servername>:<port>/<APP>/admin
Click on Application > Advanced Properties and enable the following MBeans value to "true"
- Enable Scenario Details MBean
- Enable Scenario Metrics MBean
Once the Custom Resource Intensive Scenario MBean (Example MyCustomExpensiveScenario) has been started and with the MBeans enabled, you should be able to see the MBean via RepoDebug under the domain
com.ibm.team.foundation.scenarios
https://<servername>:<port>/<APP>/repodebug/mxBeans
Please see the article
CLM Monitoring in section
6. Resource-Intensive Scenarios Summary bean for more information.
Viewing the Custom Scenario Statistics
Once the Resource Intensive Scenario MBean has been stopped the Counters will be updated. To view these Scenario Statistics open the following URL.
https://<servername>:<port>/<APP>/service/com.ibm.team.repository.service.internal.counters.ICounterContentService
The Scenarios counters can be found in the section
scenarios
The Scenarios summary information can be found in the section
scenariosSummary
Custom scenario counter and scenario summary.:
Examples of Registering a Resource-Intensive Scenario
Using Curl Script
RESTClient allows using JSESSIONID from existing session, as a value to “X-Jazz-CSRF-Prevent” header. For Curl, below command line can be used to create a valid session. Below sample further, captures the session in a variable for use with other commands
Declare a USER, PWD and COOKIES variables to begin with:
$ USER=JazzUser
$ PWD=JazzUserPassword
$ COOKIES=cookies.txt
Perform the authentication and capture the cookies
$ curl -k -c $COOKIES https://<servername>:<port>/ccm/authenticated/identity
$ curl -k -L -b $COOKIES -c $COOKIES -d j_username=$USER -d j_password=$PWD https://<servername>:<port>/ccm/authenticated/j_security_check
Start Scenario
POST to startscenario url results in marking the beginning of a new instance of the scenario.
Curl provides two ways for passing the JSON content
a. Using a JSON file representing the scenario object
$ echo {scenarioName: \"curlScenario01\"} > scenarioName.json
$ curl -X POST -k -b $COOKIES -H 'Content-Type: application/json' -H 'Accept: application/json' 'https://<servername>:<port>/ccm/service/com.ibm.team.repository.service.serviceability.IScenarioRestService/scenarios/startscenario' -d @scenarioName.json > scenarioResponse.json
b. Using a JSON Object data directly in command line
$ curl -X POST -k -b $COOKIES -H 'Content-Type: application/json' -H 'Accept: application/json' 'https://<servername>:<port>/ccm/service/com.ibm.team.repository.service.serviceability.IScenarioRestService/scenarios/startscenario' --data "{\"scenarioName\": \"curlScenario01\"}" > scenarioResponse.json
Every post creates a newer instance of the Scenario. The RESPONSE from the POST has the details of the instance.
Response from the POST above looks like this :
{"scenarioName":"curlScenario01","scenarioInstanceId":"_4fTGABQTEemlX9f2gd8Dpw","scenarioHeaderKey":"x-com-ibm-team-scenario","scenarioHeaderValue":"_4fTGABQTEemlX9f2gd8Dpw%3Bname%3DcurlScenario01"}
scenarioHeaderValue in the above response shows which scenario this instance belongs to.
Run Scenario
Invoke the requests that are part of this scenario
Here is an example request that invokes a work item query :
$ curl -X GET -k -b $COOKIES https://<servername>:<port>/ccm/oslc/contexts/_kEYLgAUhEemlX9f2gd8Dpw/workitems -H 'Accept: application/rdf+xml'
Stop Scenario
$ curl -X POST -k -b $COOKIES -H 'Content-Type: application/json' -H 'Accept: application/json' -i 'https://<servername>:<port>/ccm/service_/com.ibm.team.repository.service.serviceability.IScenarioRestService/scenarios/*stopscenario*' -d @scenarioResponse.json
Counters get updated after the scenario stops.
Updated
scenario and instance counts can be seen in the “scenarios” section of the application admin counters page, here is the URL:
https://<servername>:<port>/ccm/admin?internal#*action=com.ibm.team.repository.admin.viewCounters*
Using Java OSLC
Open source Java code is available
in the Jazz Community project custom-expensive-scenario-notifier-oslc4j.
The code can be deployed as executable JAR file and comes with scripts to start and stop an expensive scenario. The scenario information is persisted in a file which allows to run any code required for the scenario.
Cosiderations
The code in this project is based on
OSLC4J and independent of libraries shipped with the CLM products. The time it takes to execute one of the commands from a command line is short compared to the next solution based on the Plain Java Client libraries. This solution is best used in build scripts or other occasions, where the code is called from a command line. The overhead is relatively minor and the tool is easily packaged in a jar file.
- Consider using this, or the CURL based solution above in command line solutions.
- This code can also work against all CLM servers, provided the login is adjusted.
The code provides the Class
ExpensiveScenarioService
that implements the interface
IExpensiveScenarioService
which basically manage the scenario registration information as string. The start command returns the JSON information about the registered expensive scenario as string. This string needs to be passed to the stop command as parameter to stop the expensive scenario.
The Class
FilePersitentExpensiveScenarioService
that implements the interface
IPersistedExpensiveScenarioService
handles the management of the scenario registration information and persists the information as a JSON file.. The start command starts the scenario and stores the JSON information about the registered expensive scenario as file. This file is read by the stop command to stop the expensive scenario.
Syntax
The syntax is as follows
-command expensiveScenario -url https://<server>:port/<context>/ -user <userId> -password <password> -scenarioName <scenarioName> -mode <mode>
Start Scenario
To start a scenario use parameter
-command expensiveScenario -url https://clm.example.com:9443/ccm/ -user ADMIN -password **** -scenarioName MyCustomExpensiveScenario -mode start
Stop Scenario
To stop a scenario use parameter
-command expensiveScenario -url https://clm.example.com:9443/ccm/ -user ADMIN -password **** -scenarioName MyCustomExpensiveScenario -mode stop
Using The Plain Java Client Libraries
Open source Java code is available
in the Jazz Community project custom-expensive-scenario-notifier-plainjava.
The code can be deployed as executable JAR file and comes with scripts to start and stop an expensive scenario. The scenario information is persisted in a file which allows to run any code required for the scenario.
Cosiderations
The code in these projects is based on the RTC Plain Java Client Libraries shipped with the CLM products. The solution can work against all applications based on Jazz Foundation. A benefit of this version is that the code potentially works against all authentication schemes, especially Open ID Connect (OIDC). The
OSLC4J solution currently does not. The time it takes to execute one of the commands from a command line is approximately twice as long as the
OSLC4J based solution. The reason is the initialization of the
TeamPlatform. But in a situation where the
TeamPlatform is already loaded and the user login has already been performed, only the code in com.ibm.js.team.monitoring.custom.service.expensivescenario has to be run which will be very fast. This solution is best used in Java server extensions or in Plain Java Client Libraries based automation, where the login into the team repository has already been performed, the
TeamPlatform is already up and the
ITeamRepository object is already available. It can be seamlessly integrated into such a scenario.
- Consider using this, in Plain Java Client Library based extensions or automation.
- This code can supports OIDC authentication.
The code provides the Class
ExpensiveScenarioService
that implements the interface
IExpensiveScenarioService
which basically manage the scenario registration information as string. The start command returns the JSON information about the registered expensive scenario as string. This string needs to be passed to the stop command as parameter to stop the expensive scenario.
The Class
FilePersitentExpensiveScenarioService
that implements the interface
IPersistedExpensiveScenarioService
handles the management of the scenario registration information and persists the information as a JSON file.. The start command starts the scenario and stores the JSON information about the registered expensive scenario as file. This file is read by the stop command to stop the expensive scenario.
Syntax
The syntax is as follows
<publicURI> <userId> <password> mode <scenarionName>
Where mode is
start
or
stop
.
Start Scenario
To start a scenario use parameter
https://clm.example.com:9443/ccm/ ADMIN **** start MyCustomExpensiveScenario
Stop Scenario
To stop a scenario use parameter
https://clm.example.com:9443/ccm/ ADMIN **** stop MyCustomExpensiveScenario
External links: