Using Curl command line to monitor performance
I thought this might help others... no question, just helpful tips.
Using Firebug, I inspected this link:
https://ben.oncloudone.com/jts/admin#action=com.ibm.team.repository.admin.statistics
I was able to discover a XML lookup by filtering by XHR on Firebug's Net console. Here is the direct link I found:
https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.counters.ICountersRestService/allCounters
We're in business :) That's the goldmine – it shows everything in XML format, including all the things that are listed on this page:
https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.counters.ICounterContentService
However, it appears that those statistics are only generated when the "Run Diagnostics" is performed, so we need to first hit these links to kick off the diagnostics:
Here is how to reproduce the login, diagnostics run, and results lookup using curl:
Using Firebug, I inspected this link:
https://ben.oncloudone.com/jts/admin#action=com.ibm.team.repository.admin.statistics
I was able to discover a XML lookup by filtering by XHR on Firebug's Net console. Here is the direct link I found:
https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.counters.ICountersRestService/allCounters
We're in business :) That's the goldmine – it shows everything in XML format, including all the things that are listed on this page:
https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.counters.ICounterContentService
However, it appears that those statistics are only generated when the "Run Diagnostics" is performed, so we need to first hit these links to kick off the diagnostics:
- https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.jfs.indexing.service.internal.diagnostics.IndicesDiagnostics
- https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.jfs.indexing.service.internal.diagnostics.StorageDiagnostics
- https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.DBIndexDiagnostic
- https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.backgroundTaskDiagnostic
- https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.systemClockDiagnostic
- https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.database.internal.databaseDiagnostic
- https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.database.internal.databaseStatisticsDiagnostic
- https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.jvm.internal.jvmStateDiagnostic
- https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.internal.discovery.diagnostics.FriendsDiagnostic
Here is how to reproduce the login, diagnostics run, and results lookup using curl:
#Request login curl -k -c cookies.txt "https://ben.oncloudone.com/jts/authenticated/identity"
#Login
curl -k -L -b cookies.txt -c cookies.txt -d j_username=JazzAdmin -d j_password='yourpassword' "https://ben.oncloudone.com/jts/authenticated/j_security_check"
#Request Diagnostics
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.jfs.indexing.service.internal.diagnostics.IndicesDiagnostics"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.jfs.indexing.service.internal.diagnostics.StorageDiagnostics"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.DBIndexDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.backgroundTaskDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.basic.internal.systemClockDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.database.internal.databaseDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.database.internal.databaseStatisticsDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.diagnostics.jvm.internal.jvmStateDiagnostic"
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticRestService/newDiagnosticResult?id=com.ibm.team.repository.service.internal.discovery.diagnostics.FriendsDiagnostic"
#Request results
curl -k -L -b cookies.txt "https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.counters.ICountersRestService/allCounters"
6 answers
Hi Ben,
Thanks for sharing that.
You can also request the entire diagnostic .zip file by requesting https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticDetailedResultsService/. The contents contains other useful information about database and jvm statistics.
Are you doing something additional with the performance data once you've gotten it from the server (other than manually reviewing the results)?
Thanks for sharing that.
You can also request the entire diagnostic .zip file by requesting https://ben.oncloudone.com/jts/service/com.ibm.team.repository.service.internal.diagnostic.IDiagnosticDetailedResultsService/. The contents contains other useful information about database and jvm statistics.
Are you doing something additional with the performance data once you've gotten it from the server (other than manually reviewing the results)?
Yes, we're automating results into a performance monitoring solution. I'd love to know if there is a recommended set of thresholds on which to alert for all these statics? There is so much data but we're not sure what "healthy" results should look like? Also, is there a way to get the results to come back in JSON format instead of XML?
I saw that the exported diagnostic zip file has much more in it... is there a way to get those results as well in JSON/XML instead of a zip download?
Thanks!
I saw that the exported diagnostic zip file has much more in it... is there a way to get those results as well in JSON/XML instead of a zip download?
Thanks!
It seems the only results that will return XML are the counters and allDiagnosticResults (which is just the top level results you see when you run the diagnostics from jts/admin -> Diagnostics). The rest of the HTML data from the export looks to be executed natively rather than through REST. You could still parse the HTML tables, for example, the "Max Execution Time" in the database results and compare that with the average time from previous runs.
Another good place to look in the performance counter results is the "Jena RDF" sections as they'll show various information about the state of the indexer (which you can also get at a higher level by doing a GET on the jts/indexing page (XML result), noting the "backlog" sections for a high count of backlog items).
As for threshold recommendations, the best way to determine outliers is likely to just get a good baseline from previous results while the server is at its peak usage, as each environment may have drastically different results (ie - a server running all apps on Tomcat with 5 concurrent versus an enterprise topology with hundreds of concurrent users).
A more general approach would be to tune the "Jena query operation logging time threshold (in ms)" to a value lower than the default 45 seconds (45000 ms) (ie - 15000 for 15 seconds, but probably not any higher than 10 seconds), which will result in log entries to the jts.log to let you know about any queries that exceeded the execution threshold. The result would look something like what is in the log below, but not necessarily every query that shows up is "bad", but could be helpful for identifying a performance problem if you see it in the log consistently when you are noticing performance problems.
2012-05-29 23:43:19,546 [ http-9443-Processor18] WARN com.ibm.team.jfs - CRJZS5546W Query _g2mdcKoJEeG1ZKu_hF3Tlw had a wait time of 0 ms, an execution time of 6147 ms, and produced 100 results
2012-05-29 23:44:08,209 [ http-9443-Processor20] WARN com.ibm.team.jfs - CRJZS5541W Query _tYssEKoJEeG1ZKu_hF3Tlw has exceeded the configurable warning threshold of 5 seconds. It has 0 user contexts:
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX jazz: <http://jazz.net/xmlns/foundation/1.0/>
PREFIX dc: <http://purl.org/dc/terms/>
SELECT ?resource ?sameAs
WHERE
{ ?resource dc:isPartOf <https://clmweb.ibm.com:9443/rm/types> ;
jazz:resourceContext <https://clmweb.ibm.com:9443/jts/process/project-areas/_rc_0YKoJEeG1ZKu_hF3Tlw> ;
owl:sameAs ?sameAs .
}
Hope it helps!
Another good place to look in the performance counter results is the "Jena RDF" sections as they'll show various information about the state of the indexer (which you can also get at a higher level by doing a GET on the jts/indexing page (XML result), noting the "backlog" sections for a high count of backlog items).
As for threshold recommendations, the best way to determine outliers is likely to just get a good baseline from previous results while the server is at its peak usage, as each environment may have drastically different results (ie - a server running all apps on Tomcat with 5 concurrent versus an enterprise topology with hundreds of concurrent users).
A more general approach would be to tune the "Jena query operation logging time threshold (in ms)" to a value lower than the default 45 seconds (45000 ms) (ie - 15000 for 15 seconds, but probably not any higher than 10 seconds), which will result in log entries to the jts.log to let you know about any queries that exceeded the execution threshold. The result would look something like what is in the log below, but not necessarily every query that shows up is "bad", but could be helpful for identifying a performance problem if you see it in the log consistently when you are noticing performance problems.
2012-05-29 23:43:19,546 [ http-9443-Processor18] WARN com.ibm.team.jfs - CRJZS5546W Query _g2mdcKoJEeG1ZKu_hF3Tlw had a wait time of 0 ms, an execution time of 6147 ms, and produced 100 results
2012-05-29 23:44:08,209 [ http-9443-Processor20] WARN com.ibm.team.jfs - CRJZS5541W Query _tYssEKoJEeG1ZKu_hF3Tlw has exceeded the configurable warning threshold of 5 seconds. It has 0 user contexts:
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX jazz: <http://jazz.net/xmlns/foundation/1.0/>
PREFIX dc: <http://purl.org/dc/terms/>
SELECT ?resource ?sameAs
WHERE
{ ?resource dc:isPartOf <https://clmweb.ibm.com:9443/rm/types> ;
jazz:resourceContext <https://clmweb.ibm.com:9443/jts/process/project-areas/_rc_0YKoJEeG1ZKu_hF3Tlw> ;
owl:sameAs ?sameAs .
}
Hope it helps!
OK - so continuing, let's say you wanted to capture the resulting XML document from allCounters and wanted to parse the output. The easiest way I found was to use XPATH to find each of the individual data values... all 2911 of them!
Here is a quick and dirty python script which I used to determine all the XPATH's:
The next step is to use all these xpath's to extract the data and start plotting these into graphs in LogicMonitor.com. Finally, I hope to set alerts on various thresholds for some of these values.
Here is a quick and dirty python script which I used to determine all the XPATH's:
import sys
file = open("output.xml")
group = ""
counter = ""
facet = ""
datavalue = ""
count = 0
foundGroup = False
foundCounter = False
foundFacet = False
foundDataValue = False
for line in file:
line = line.strip()
if (foundGroup):
foundGroup=False
group = line.split("")[1].split(" ")[0]
#print "Group = " + group
continue
if (foundCounter):
foundCounter = False
counter = line.split("")[1].split(" ")[0]
#print "Counter = " + counter
continue
if (foundFacet):
foundFacet = False
foundDataValue = True
facet = line.split("")[1].split(" ")[0]
#print "Facet = " + facet
continue
if (foundDataValue):
if(line==""):
foundDataValue = False
continue
datavalue = line.split("<")[1].split(">")[0]
#print "Data Value = " + datavalue
print group + " - " + counter + " - " + facet + " = " + "//groups[name=\"" + group + "\"]/counters[name=\"" + counter + "\"]/facets[name=\"" + facet + "\"]/" + datavalue
count+=1
continue
if(line==""):
foundGroup = True
continue
if(line==""):
foundCounter = True
continue
if(line==""):
foundFacet = True
continue
file.close()
print "Count = " + str(count)
I would paste all these XPATH's but I think someone would get angry!
The next step is to use all these xpath's to extract the data and start plotting these into graphs in LogicMonitor.com. Finally, I hope to set alerts on various thresholds for some of these values.
Ben - thanks your curl login example solved a problem for me.
I've adapted these to Powershell 4.0, which for the login seems to need the UserAgent set to something different than the default, and NOTE the accept parameter is " * / * " (without the spaces) which doesn't seem to show up properly on here:
ALSO I don't know what the o:p tags are in the commands below, jazz.net seems to be adding them all by itself and you shouldn't include them in the Powershell command..
Request Login
-
Invoke-WebRequest -SessionVariable cookies -Uri https://ian.myserver.com/jts/authenticated/identity <o:p> </o:p>
Actually login:
-
Invoke-WebRequest -WebSession $cookies -Headers @{"accept"="/"} -Method Post -Body @{"j_username"="userid";"j_password"="pass"} -Uri "https://ian.myserver.com/jts/authenticated/j_security_check" -UserAgent "curl" <o:p> </o:p>
And then for all subsequent requests include -WebSession $cookies in the Invoke-WebRequest, and if expecting data then you probably need the Accept header (value " * / * " without the spaces), like this:
Retrieve project catalog
-
Invoke-WebRequest -WebSession $cookies -Headers @{"accept"="/"} -OutFile projects.xml -Uri "https://ian.myserver.com/rm/oslc_rm/catalog" <o:p> </o:p>