It's all about the answers!

Ask a question

Using Curl command line to monitor performance


Benjamin Chodroff (8985131) | asked Jul 16 '12, 7:15 p.m.
edited Jul 16 '12, 10:03 p.m.
I thought this might help others... no question, just helpful tips.

Using Firebug, I inspected this link:
https://ben.oncloudone.com/jts/admin#action=com.ibm.team.repository.admin.statistics

I was able to discover a XML lookup by filtering by XHR on Firebug's Net console. Here is the direct link I found:
https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.counters.ICountersRestService/allCounters

We're in business :) That's the goldmine – it shows everything in XML format, including all the things that are listed on this page:
https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.counters.ICounterContentService

However, it appears that those statistics are only generated when the "Run Diagnostics" is performed, so we need to first hit these links to kick off the diagnostics:
  1. https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.jfs.indexing.service.internal.diagnostics.IndicesDiagnostics
  2. https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.jfs.indexing.service.internal.diagnostics.StorageDiagnostics
  3. https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.DBIndexDiagnostic
  4. https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.backgroundTaskDiagnostic
  5. https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.systemClockDiagnostic
  6. https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.database.internal.databaseDiagnostic
  7. https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.database.internal.databaseStatisticsDiagnostic
  8. https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.jvm.internal.jvmStateDiagnostic
  9. https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.internal.discovery.diagnostics.FriendsDiagnostic

Here is how to reproduce the login, diagnostics run, and results lookup using curl:

#Request login


curl -k -c cookies.txt "https://ben.oncloudone.com/jts/authenticated/identity"

#Login
curl -k -L -b cookies.txt -c cookies.txt -d j_username=JazzAdmin -d j_password='yourpassword' "https://ben.oncloudone.com/jts/authenticated/j_security_check"
#Request Diagnostics
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.jfs.indexing.service.internal.diagnostics.IndicesDiagnostics"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.jfs.indexing.service.internal.diagnostics.StorageDiagnostics"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.DBIndexDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.backgroundTaskDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.systemClockDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.database.internal.databaseDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.database.internal.databaseStatisticsDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.jvm.internal.jvmStateDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.internal.discovery.diagnostics.FriendsDiagnostic"
#Request results
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.counters.ICountersRestService/allCounters"

6 answers



permanent link
Paul Slauenwhite (8.3k12) | answered Jul 17 '12, 6:18 a.m.
FORUM MODERATOR / JAZZ DEVELOPER
Noteworthy, these APIs are internal.  As such, they are subject to change at any time.

permanent link
Benjamin Silverman (4.1k610) | answered Jul 16 '12, 8:15 p.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER
Hi Ben,

Thanks for sharing that. 

You can also request the entire diagnostic .zip file by requesting https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticDetailedResultsService/.  The contents contains other useful information about database and jvm statistics.

Are you doing something additional with the performance data once you've gotten it from the server (other than manually reviewing the results)?

permanent link
Benjamin Chodroff (8985131) | answered Jul 16 '12, 9:57 p.m.
Yes, we're automating results into a performance monitoring solution. I'd love to know if there is a recommended set of thresholds on which to alert for all these statics? There is so much data but we're not sure what "healthy" results should look like? Also, is there a way to get the results to come back in JSON format instead of XML?

I saw that the exported diagnostic zip file has much more in it... is there a way to get those results as well in JSON/XML instead of a zip download?

Thanks!

permanent link
Benjamin Silverman (4.1k610) | answered Jul 17 '12, 12:22 a.m.
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER
It seems the only results that will return XML are the counters and allDiagnosticResults (which is just the top level results you see when you run the diagnostics from jts/admin -> Diagnostics).  The rest of the HTML data from the export looks to be executed natively rather than through REST.  You could still parse the HTML tables, for example, the "Max Execution Time" in the database results and compare that with the average time from previous runs.

Another good place to look in the performance counter results is the "Jena RDF" sections as they'll show various information about the state of the indexer (which you can also get at a higher level by doing a GET on the jts/indexing page (XML result), noting the "backlog" sections for a high count of backlog items).

As for threshold recommendations, the best way to determine outliers is likely to just get a good baseline from previous results while the server is at its peak usage, as each environment may have drastically different results (ie - a server running all apps on Tomcat with 5 concurrent versus an enterprise topology with hundreds of concurrent users). 

A more general approach would be to tune the "Jena query operation logging time threshold (in ms)" to a value lower than the default 45 seconds (45000 ms) (ie - 15000 for 15 seconds, but probably not any higher than 10 seconds), which will result in log entries to the jts.log to let you know about any queries that exceeded the execution threshold.  The result would look something like what is in the log below, but not necessarily every query that shows up is "bad", but could be helpful for identifying a performance problem if you see it in the log consistently when you are noticing performance problems.

2012-05-29 23:43:19,546 [         http-9443-Processor18]  WARN com.ibm.team.jfs                                    - CRJZS5546W Query _g2mdcKoJEeG1ZKu_hF3Tlw had a wait time of 0 ms, an execution time of 6147 ms, and produced 100 results
2012-05-29 23:44:08,209 [         http-9443-Processor20]  WARN com.ibm.team.jfs                                    - CRJZS5541W Query _tYssEKoJEeG1ZKu_hF3Tlw has exceeded the configurable warning threshold of 5 seconds. It has 0 user contexts:
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX jazz: <http://jazz.net/xmlns/foundation/1.0/>
PREFIX dc: <http://purl.org/dc/terms/>
SELECT ?resource ?sameAs
WHERE
{ ?resource dc:isPartOf <https://clmweb.ibm.com:9443/rm/types> ;
jazz:resourceContext <https://clmweb.ibm.com:9443/jts/process/project-areas/_rc_0YKoJEeG1ZKu_hF3Tlw> ;
owl:sameAs ?sameAs .
}

Hope it helps!

permanent link
Benjamin Chodroff (8985131) | answered Jul 26 '12, 7:35 p.m.
edited Jul 26 '12, 7:37 p.m.
OK - so continuing, let's say you wanted to capture the resulting XML document from allCounters and wanted to parse the output. The easiest way I found was to use XPATH to find each of the individual data values... all 2911 of them!

Here is a quick and dirty python script which I used to determine all the XPATH's:

import sys

file = open("output.xml")
group = ""
counter = ""
facet = ""
datavalue = ""
count = 0

foundGroup = False
foundCounter = False
foundFacet = False
foundDataValue = False

for line in file:
        line = line.strip()
        if (foundGroup):
                foundGroup=False
                group = line.split("")[1].split("")[0]
                #print "Group = " + group
                continue
        if (foundCounter):
                foundCounter = False
                counter = line.split("")[1].split("")[0]
                #print "Counter = " + counter
                continue
        if (foundFacet):
                foundFacet = False
                foundDataValue = True
                facet = line.split("")[1].split("")[0]
                #print "Facet = " + facet
                continue
        if (foundDataValue):
                if(line==""):
                        foundDataValue = False
                        continue
                datavalue = line.split("<")[1].split(">")[0]
                #print "Data Value = " + datavalue
                print group + " - " + counter + " - " + facet + " = " + "//groups[name=\"" + group + "\"]/counters[name=\"" + counter + "\"]/facets[name=\"" + facet + "\"]/" + datavalue
                count+=1
                continue
        if(line==""):
                foundGroup = True
                continue
        if(line==""):
                foundCounter = True
                continue
        if(line==""):
                foundFacet = True
                continue
file.close()

print "Count = " + str(count)

I would paste all these XPATH's but I think someone would get angry!
The next step is to use all these xpath's to extract the data and start plotting these into graphs in LogicMonitor.com. Finally, I hope to set alerts on various thresholds for some of these values.


permanent link
Ian Barnard (960612) | answered Mar 14 '18, 6:31 a.m.
JAZZ DEVELOPER
edited Mar 23 '18, 6:43 a.m.

Ben - thanks your curl login example solved a problem for me.

I've adapted these to Powershell 4.0, which for the login seems to need the UserAgent set to something different than the default, and NOTE the accept parameter is " * / * " (without the spaces) which doesn't seem to show up properly on here:

ALSO I don't know what the o:p tags are in the commands below, jazz.net seems to be adding them all by itself and you shouldn't include them in the Powershell command..

Request Login

Actually login:

And then for all subsequent requests include -WebSession $cookies in the Invoke-WebRequest, and if expecting data then you probably need the Accept header (value " * / * " without the spaces), like this:

Retrieve project catalog

Your answer


Register or to post your answer.