(r3) LifecycleQueryEnginePerformanceReportV602 < Deployment

~~EditAttach~~Printable
r3 - 2016-03-16 - 14:35:23 - Main.gsvenYou are here: TWiki >
Deployment Web > DeploymentPlanningAndDesign > PerformanceDatasheetsAndSizingGuidelines > LifecycleQueryEnginePerformanceReportV602
<div id="header-title" style="padding: 10px 15px; border-width:1px; border-style:solid; border-color:#FFD28C; background-image: url(<nop>https://jazz.net/wiki/pub/Deployment/WebPreferences/TLASE.jpg); background-size: cover; font-size:120%">
---+!! Lifecycle Query Engine 6.0.2 Performance Report for Query Scalability <img src="https://jazz.net/wiki/pub/Deployment/WebPreferences/uc.png" alt="uc.png" width="50" height="50" align="right"> 
%DKGRAY% Authors: Gustaf Svensson<br>
Build basis: Lifecycle Query Engine 6.0.2
%ENDCOLOR%</div></sticky>

<!-- Page contents top of page on right hand side in box -->
<sticky><div style="float:right; border-width:1px; border-style:solid; border-color:#DFDFDF; background-color:#F6F6F6; margin:0 0 15px 15px; padding: 0 15px 0 15px;">
%TOC{title="Page contents"}%
</div></sticky>

<sticky><div style="margin:15px;"></sticky>

---++ Introduction
This article presents the results of Lifecycle Query Engine (LQE) performance testing for the Rational Collaborative Lifecycle Managment (CLM) 6.0.2 release. JRS and LQE provide an integrated view of artifacts across data sources allowing the capability to generate reports across tools and project areas.

*Note: this page is under construction and some information may not be correct*

The Lifecycle Query Engine (LQE) implements a Linked Lifecycle Data Index over data provided by one or more lifecycle tools. A lifecycle tool makes its data available for indexing by exposing its Linked Lifecycle Data via a Tracked Resource Set, whose members MUST be retrievable resources with RDF representations, called Index Resources.

An LQE Index built from one or more Tracked Resource Sets allows SPARQL queries  to be run against the RDF dataset that aggregates the RDF graphs of the Index Resources. This permits data from multiple lifecycle tools to be queried together, including cross-tool links between resources. Changes that happen to Index Resources in a lifecycle tool are made discoverable via the Tracked Resource Set's Change Log, allowing the changes to be propagated to the Lifecycle Index to keep it up to date.

JRS provides a Report Builder to guide users through the intricacies of building SPARQL ueries to view data in a report format. 
This article will show performance benchmarks from LQE tests providing guidance to customers in planning their LQE deployment and server configurations.

---+++!! Disclaimer

%INCLUDE{"PerformanceDatasheetDisclaimerEndToEnd"}%

---++ Test details

---+++ What Our Tests Measure
*Automated tooling such as Rational Performance Tester (RPT) is widely used to simulate a workload normally generated by client software such as the Eclipse client or web browsers. All response times listed are those measured by our automated tooling and not a client. 

The diagram below describes at a very high level which aspects of the entire end-to-end experience (human end-user to server and back again) that our performance tests simulate. The tests described in this article simulate a segment of the end-to-end transaction as indicated in the middle of the diagram. Performance tests are server-side and capture response times for this segment of the transaction. *

---+++ Topology

The topology under test is based on [[RecommendedCLMDeploymentTopologies5#CLM_E1_Enterprise_Distributed_Li][Standard Topology (E1) Enterprise - Distributed / Linux / DB2.]]

<img src="%ATTACHURLPATH%/LQETopology2.png" alt="Server Overview" width="60%" height="60%" />

The specifications of machines under test are listed in the table below. Server tuning details listed in *[[#AppendixA][Appendix A]]*

<table class="gray-table">
	<tbody>
		<tr>
			<th valign="top">Function</th>
			<th valign="top">Number of Machines</th>
			<th valign="top">Machine Type</th>
			<th valign="top">CPU / Machine</th>
			<th valign="top">Total # of CPU Cores/Machine</th>
			<th valign="top">Memory/Machine</th>
			<th valign="top">Disk</th>
			<th valign="top">Disk capacity</th>
			<th valign="top">Network interface</th>
			<th valign="top">OS and Version</th>
		</tr>
		<tr>
			<td>Proxy Server (IBM HTTP Server and !WebSphere Plugin)</td>
			<td>1</td>
			<td>IBM System x3250 M4</td>
			<td>1 x Intel Xeon E3-1240 3.4GHz (quad-core)</td>
			<td>8</td>
			<td>16GB</td>
			<td>RAID 1 -- SAS Disk x 2</td>
			<td>279GB</td>
			<td>Gigabit Ethernet</td>
			<td>Red Hat Enterprise Linux Server release 6.3 (Santiago)</td>
		</tr>
		<tr>
			<td>JTS Server</td>
			<td>1</td>
			<td>IBM System x3550 M4</td>
			<td>2 x Intel Xeon E5-2640 2.5GHz (six-core)</td>
			<td>24</td>
			<td>32GB</td>
			<td>RAID 5 -- SAS Disk x 4</td>
			<td>279GB</td>
			<td>Gigabit Ethernet</td>
			<td>Red Hat Enterprise Linux Server release 6.3 (Santiago)</td>
		</tr>
		<tr>
			<td>RDNG Server</td>
			<td>1</td>
			<td>IBM System x3550 M4</td>
			<td>2 x Intel Xeon E5-2640 2.5GHz (six-core)</td>
			<td>24</td>
			<td>32GB</td>
			<td>RAID 5 -- SAS Disk x 4</td>
			<td>279GB</td>
			<td>Gigabit Ethernet</td>
			<td>Red Hat Enterprise Linux Server release 6.3 (Santiago)</td>
		</tr>
		<tr>
			<td>Database Server</td>
			<td>1</td>
			<td>IBM System x3650 M4</td>
			<td>2 x Intel Xeon E5-2640 2.5GHz (six-core)</td>
			<td>24</td>
			<td>64GB</td>
			<td>RAID 10 -- SAS Disk x 16</td>
			<td>279GB</td>
			<td>Gigabit Ethernet</td>
			<td>Red Hat Enterprise Linux Server release 6.3 (Santiago)</td>
		</tr>
		<tr>
			<td>RPT Workbench</td>
			<td>1</td>
			<td>VM image</td>
			<td>2 x Intel Xeon X7550 CPU (1-Core 2.0GHz 64-bit)</td>
			<td>2</td>
			<td>6GB</td>
			<td>SCSI</td>
			<td>80GB</td>
			<td>Gigabit Ethernet</td>
			<td>Microsoft Windows Server 2003 R2 Standard Edition SP2</td>
		</tr>
		<tr>
			<td>RPT Agent</td>
			<td>1</td>
			<td>xSeries 345</td>
			<td>4 x Intel Xeon X3480 CPU (1-Core 3.20GHz 32-bit)</td>
			<td>4</td>
			<td>3GB</td>
			<td>SCSI</td>
			<td>70GB</td>
			<td>Gigabit Ethernet</td>
			<td>Microsoft Windows Server 2003 Enterprise Edition SP2</td>
		</tr>
		<tr>
			<td>RPT Agent</td>
			<td>1</td>
			<td>xSeries 345</td>
			<td>4 x Intel Xeon X3480 CPU (1-Core 3.20GHz 32-bit)</td>
			<td>4</td>
			<td>3GB</td>
			<td>RAID 1 - SCSI Disk x 2</td>
			<td>70GB</td>
			<td>Gigabit Ethernet</td>
			<td>Microsoft Windows Server 2003 Enterprise Edition SP2</td>
		</tr>
		<tr>
			<td>RPT Agent</td>
			<td>1</td>
			<td>Lenovo 9196A49</td>
			<td>1 x Intel Xeon E6750 CPU (2-Core 2.66GHz 32-bit)</td>
			<td>2</td>
			<td>2GB</td>
			<td>SATA</td>
			<td>230GB</td>
			<td>Gigabit Ethernet</td>
			<td>Microsoft Windows Server 2003 Enterprise Edition SP2</td>
		</tr>
		<tr>
			<td>Network switches</td>
			<td>N/A</td>
			<td>Cisco 2960G-24TC-L</td>
			<td>N/A</td>
			<td>N/A</td>
			<td>N/A</td>
			<td>N/A</td>
			<td>N/A</td>
			<td>Gigabit Ethernet</td>
			<td>24 Ethernet 10/100/1000 ports</td>
		</tr>
	</tbody>
</table>



---+++Network connectivity

All server machines and test clients are located on the same subnet. The LAN has 1000 Mbps of maximum bandwidth and less than 0.3ms latency in ping.




---+++ Methodology
LQE used the JMeter performance measurement tool from Apache to simulate clients as JMeter Threads. Each user is represented by 1 JMeter thread.  For these tests, the load is incremented by 50 threads (users) between 50 and 650 (or higher).

The performance test models a dashboard refresh initiated by a user. When a user refreshes a dashboard, each widget  that refreshes represents one or more queries which get executed in parallel. 

Dashboard widgets can represent fast user-related queries, slower, larger scoped queries called report queries, or more encompassing project-level queries. In these tests, about 80% of the queries are classified as user-related queries, and the rest are report-based queries.

Each thread (user) refreshes their dashboard resulting in 6 or more queries executed at the same time.  The total number of threads are ramped up over a period of 5 minutes. Then each thread will continue to refresh their dashboard 15 times with a think time of 80 seconds between refreshes. 

Between concurrency tests there is 2.67 minute pause to allow the system to quiesce.
Queries are tested against two different 100M triple datasets:
   * Reslqe – the reslqe dataset is a generated dataset which was seeded by a real LQE indexTdb which had indexed 15 internal IBM RTC projects. The dataset was profiled and  then increased in size to 100M triples by an internal data generator utility.
   * SSE – The SSE dataset started as a RELM LQE indexTdb which was profiled and grown in size to 100M triples by an internal data generator utility.

The reslqe queries consisted of 8 queries: 2 report based queries and 6 user based queries.

|  *Run*  |  *Total Queries Executed*  |
|  run_50  |  6000  |
|  run_100  |  12000  |
|  run_150  |  18000  |
|  run_200  |  24000  |
|  run_250  |  30000  |
|  run_300  |  36000  |
|  run_350  |  42000  |
|  run_400  |  48000  |
|  run_450  |  54000  |
|  run_500  |  60000  |
|  run_550  |  66000  |
|  run_600  |  72000  |
|  run_650  |  78000  |
Table 1 Total Number of reslqe queries per run

The SSE queries consisted of 10 queries: 4 report based queries and  6 user based queries,
|  *Run*  |  *Total Queries Executed*  |
|  run_50  |  7500  |
|  run_100  |  15000  |
|  run_150  |  22500  |
|  run_200  |  30000  |
|  run_250  |  37500  |
|  run_300  |  45000  |
|  run_350  |  52500  |
|  run_400  |  60000  |
|  run_450  |  67500  |
|  run_500  |  75000  |
|  run_550  |  82500  |
|  run_600  |  90000  |
|  run_650  |  97500  |
Table 2 Total Number of SSE queries per run

---++ LQE Test Dataset Characteristics
|  *Condition*  |  *Reslqe*  |  *SSE*  |
| Size of  the dataset on disk, on the LQE server | 22.2GB | 19.9GB |
| Number of resources  | 3,050,000 | 4,953,453 |
| Number of triples | 99,389,595 | 99,919,403 |

---++ Performance goals 
The goal of this performance testing was to determine how many users LQE could support in a query-only based scenario when the users refreshed their dashboards throughout the day.

Customer expectations for LQE query and indexing performance should be analyzed with attention to server hardware, dataset size, data model complexity, current system load, and query optimization.  
   * Larger indexes requires more RAM native memory.
   * Heavier query loads should allocate more CPU cores.
   * Indexing can perform faster with faster disk I/O.

These tests were performed on a Redhat 7.0 server system with 96GB RAM, 250 GB SSD, 2 Intel Xeon E7-2830 8-core 2133 MHz CPUs, 100M triple datasets are based on CLM resources, and a set of dashboard-like queries.

---++ Topology 
The topology under test is based on [[https://jazz.net/wiki/bin/view/Deployment/RecommendedCLMDeploymentTopologies#CLM_E1_Enterprise_Distributed_Li][Standard Topology (E1) Enterprise - Distributed / Linux / DB2.]]

     <img src="%ATTACHURLPATH%/Topology.png" alt="Topology.png" width="601" height="482" />

You can run several instances of Lifecycle Query Engine (LQE) and each instance must use the same external data source (in this case DB2). The group of LQE nodes behave like a single logical unit. By deploying LQE across a set of servers, you can distribute the query workload and improve performance and scalability. Each LQE node contains its own independent triple store index, which indexes the same tracked resource set (TRS) data providers that you specified on the LQE Administration page.

The specifications of machines under test are listed in the table below. Server tuning details are listed in Appendix A 

|   *Role*   |   *Model*   |   *Processor Type*   |   *Number of Processors*   |   *Memory(GB)*   |   *Disk*   |   *Disk Capacity*   |   *OS*   |
| LQE 1 Server | IBM X3690-X5 | 2 x Intel Xeon E7-2830 8 core 2.13GHz | 32 vCPU | 96GB | 400GB SSD 3 x 146 GB 15k RPM SAS RAID-0 4 x 500 GB 7200 RPM SAS Raid-5 | 2.78TB | Redhat Enterprise 7 .0 |
| LQE 2 Server | IBM X3690-X5 | 2 x Intel Xeon E7-2830 8 core 2133 MHz | 32 vCPU | 96 GB | 1 x 250GB SSD 4 x 300GBV SAS RAID-5 Configuration | 1300GB | Redhat Enterprise 7.0 |
| Test Framework/DB2 Server | IBM X3690-X5 | 2 x Intel Xeon E7-2830 8 core 2133 MHz | 32 vCPU | 96 GB | 1 x 250GB SSD 4 x 300GBV SAS RAID-5 Configuration | 1072GB | Windows Server 2008 R2 Enterprise |

---+++ Disks 

For a larger configuration, the use of fibre attached SAN storage is recommended but not required. For LQE servers you should consider using SSD drives, but it depends whether your environment will easily support them. 

---+++ Network Connectivity

There are several aspects to network connectivity for LQE: 

  * Connectivity between the LQE and a database server.

  * Connectivity between the LQE server and the end users.

  * Connectivity between the LQE server and data source provider servers.

  * Connectivity between LQE and an authenticating server (e.g. JTS).
 
The recommendation for network connectivity is to minimize latency between the LQE application server and other servers (no more than 1-2 ms) and to have the servers located on the same subnet. When using external storage solutions, the recommendation is to minimize connectivity latencies, with the optimal configuration being fiber attached Storage Area Network (SAN) storage. 

---++ Results
The following results are based on LQE query performance with no active indexing.

     <img src="%ATTACHURLPATH%/Results-1-LQE-Node.png" alt="Results-1-LQE-Node.png" width="666" height="268" />

Illustration 1: Average Dashboard Response Time (1 LQE Node)

In illustration 1, the average dashboard response time exceeded 1 second with concurrent loads of 400, 450, 600 and 650 users.  In this test, there was a think time of 80 secs between dashboard refreshes.  When the refresh think time is expanded to 300 secs (5 mins) and the thread rampup period is increased, these spikes decrease and level out indicating that LQE will quiesce with higher loads when given the opportunity. It should also be noted that for all runs the average dashboard response time never exceeded 60 secs which most dashboard users would find acceptable.
The spikes in this graph usually indicate contention of either network or Jena resources. Query throughput drops with high contention rates.

     <img src="%ATTACHURLPATH%/Results-2-LQE-Node.png" alt="Results-2-LQE-Node.png" width="666" height="268" />

Illustration 2: Average Dashboard Response Time (2 LQE Node)

Illustration 2 shows the average dashboard response time for 2 LQE nodes with the same 100M triple index.  The average dashboard response time exceeded 1 second at 950 concurrent users, yet remained less that 30 seconds up to 1000 concurrent users. As can be seen in illustrations 1 and 2, a two LQE node system can expand the query capacity in a system. In illustration 1 we see 1 LQE node supports 350 users with average dashboard response times less than 1 sec.  Whereas 900 concurrent users in the 2 LQE node system are supported until contention issues begin.  Also, in the 2 LQE node system the maximum time spent during any test run was 27 seconds as compared to 43 seconds for 1 LQE node. Using a load balancer to share queries across LQE nodes helps maintain query performance expectations to support almost twice as many users. Multi-node LQE systems share the burden in a high throughput query environment.

In previous performance test experiments, it was found that query performance can degrade when the index has about 100M triples or above, but this depends on resource shapes and size of resources.


---++ Appendix A
#AppendixA
<table class="gray-table">
	<tbody>
		<tr>
			<th align="left" width="200"><strong>Product</strong><br></th>
			<th align="left" width="100"><strong>Version</strong></th>
			<th align="left" width="600"><strong>Highlights for configurations under test</strong></th>
		</tr>
		<tr>
			<td style="vertical-align: top;">IBM HTTP Server for !WebSphere Application Server</td>
			<td style="vertical-align: top;">8.5.5.1</td>
			<td style="vertical-align: top;">IBM HTTP Server functions as a reverse proxy server implemented
via Web server plug-in for !WebSphere Application Server. Configuration details can be found from the [[http://pic.dhe.ibm.com/infocenter/clmhelp/v4r0/index.jsp?topic=%2Fcom.ibm.jazz.install.doc%2Ftopics%2Ft_config_reverse_proxy_ihs.html][CLM infocenter]].

*HTTP server (httpd.conf)*:

   * !MaxClients: increase value for high-volume loads [[http://publib.boulder.ibm.com/httpserv/ihsdiag/ihs_performance.html][(adjust value based on user load)]]
   * !ThreadsPerChild = 50

*Web server plugin-in (plugin-cfg.xml)*:

   * !ServerIOTimeout="900"

*OS Configuration*:

   * max user processes = unlimited
      </td>
		</tr>
		<tr>
			<td style="vertical-align: top;">IBM !WebSphere Application Server Network Deployment</td>
			<td style="vertical-align: top;">8.5.5.1</td>
			<td style="vertical-align: top;"><strong>JVM settings:</strong>

   * GC policy and arguments, max and init heap sizes:

				<verbatim>
-XX:MaxDirectMemorySize=1g -Xgcpolicy:gencon -Xmx16g -Xms16g -Xmn4g -Xcompressedrefs -Xgc:preferredHeapBase=0x100000000 -Xverbosegclog:logs/gc.log
</verbatim>

				*Thread pools:*

   * Maximum !WebContainer = 500
   * Minimum !WebContainer = 500


				*OS Configuration:*

				System wide resources for the app server process owner: 
   * max user processes = unlimited
   * open files = 65536

			</td>
		</tr>
		<tr>
			<td>DB2</td>
			<td>DB2 10.1</td>
			<td> </td>
		</tr>
		<tr>
			<td>LDAP server</td>
			<td> </td>
			<td> </td>
		</tr>
		<tr>
			<td>License server</td>
			<td> </td>
			<td>Hosted locally by JTS server</td>
		</tr>
		<tr>
			<td>RPT workbench</td>
			<td>8.3</td>
			<td>Defaults</td>
		</tr>
		<tr>
			<td>RPT agents</td>
			<td>8.5.1</td>
			<td>Defaults</td>
		</tr>
		<tr>
			<td>Network</td>
			<td> </td>
			<td>Shared subnet within test lab</td>
		</tr>
	</tbody>
</table>


---++ Appendix AB - Server Tuning Details 

|   *Product*   |   *Version*   |   *Highlights for configurations under test*   |
|  LQE  |  LQE 6.0  |*LQE Configuration Changes:* |
|^|^| On the Query Service Page,|
|^|^| * Set the Query Timeout(seconds) to 160 |
|^|^| * Enable HTTP Response Caching Enabled and |
|^|^| * Enable Query Result Caching Enabled |
|^|^| On the Permissions Page, |
|^|^| * Set “Everyone” as the group for the Lifecycle Query Engine Index.  |
|^|^| On the Advanced Properties Page, show internal, |
|^|^| * Set the HTTP Connection Timeouts to 120 secs |
|^|^| * Set the HTTP Socket Timeouts to 120 secs |
|^|^| * Set "Maximum Number of Query Metrics" to “5000” |
|^|^| * Set "Metrics Maximum Age(ms)" to "604800000" (7 days) |
|^|^| * Set Metrics Scrubber Period (ms) to “32400000” (9 hrs) |
|^|^| * Set "Metrics Graph Count Period (seconds)" to "60000" (16.67 hrs) |
|  Apache Tomcat   |  CLM packaged Version 7.0.54  | *JVM settings:*   |
|^|^| *GC policy, arguments, max and init heap sizes:*  |
|^|^| _JAVA_OPTS="$JAVA_OPTS -Xmx16G"_ |
|^|^| _JAVA_OPTS="$JAVA_OPTS -Xms4G"_  |
|^|^| _JAVA_OPTS="$JAVA_OPTS -Xmn512M"_  |
|^|^| _JAVA_OPTS="$JAVA_OPTS -Xgcpolicy:gencon"_  |
|^|^| _JAVA_OPTS="$JAVA_OPTS -Xcompressedrefs -Xgc:preferredHeapBase=0x100000000"_  |
|^|^| _JAVA_OPTS="$JAVA_OPTS -agentlib:healthcenter -Dcom.ibm.java.diagnostics.healthcenter.agent.port=1972"_  |
|^|^| _JAVA_OPTS="$JAVA_OPTS -Dlqe.testDiscoverResourceGroups=true"_  |
|^|^| _JAVA_OPTS="$JAVA_OPTS -Dlqe.performance.test=true"_  |
|^|^| *Linux Settings:*  |
|^|^| Set Linux scaling governor to “performance” --\
  _for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do [ -f $CPUFREQ ] continue; echo -n performance > $CPUFREQ; done_ |
|  DB2  |  DB2 v10.5.300.128  |*Transaction log setting of LQE Database:*  |
|^|^|* Transaction log size changed to 8192: _db2 update db cfg using LOGFILSIZ 8192_  |
|^|^|* Transaction log primary file number to 24: _db2 update db cfg using LOGPRIMARY 24_  |
|^|^|* Transaction log secondary file number to 24: _db2 update db cfg using LOGSECOND 24_  | 
|  Network  | |Shared subnet within test lab  |

---++ Appendix B – Considerations
  * Generally, the smaller the index the faster queries will perform

  * Plan ahead, add more RAM as the indexTdb size increases

When performance expectations start to deteriorate, or when queries start to time out you should investigate the following:

  * Is compaction of the indexTdb required?

  * Is there a rogue query? ( a rogue query is a query that is not being timed out as it should be)
 
          &nbsp; A rogue query can be detected by navigating to the LQE UI Health Monitoring page: 

          &nbsp;  * Select  *Statistics*

          &nbsp;  * Select *View As List* (in the upper right hand side) 

          &nbsp;  * Select *Running Queries* tab to identify any queries which may not have timed out

  * Has the IndexTdb grown to surpass the server hardware capacities such as the allocated RAM?

  * Is the system under too high a load? If queries are continuous, indexing may not have enough opportunity to write to the index.

  * Queries which are run frequently and that take longer than 1 second to execute should be optimized.

  * Query performance might decrease when more than 3000 results are returned.  (ie 150,000 results may take more time)

  * The query timeout is set to 60 seconds as a default:  If queries are taking longer, increase the query timeout time on the Query Services page, or analyze the system for other possible concerns such as those listed above.

  * Do not copy an indexTdb directory when the LQE server is active.

---+++++!! Additional contributors: Main.TWikiUser, Main.TWikiUser

<sticky></div></sticky>
Attachments
Topic attachments
I	Attachment	Action	Size	Date	Who	Comment
png	LQETopology.png	manage	23.4 K	2016-03-15 - 20:58	UnknownUser
png	LQETopology2.png	manage	28.4 K	2016-03-16 - 14:13	UnknownUser
Deployment
Community information and contribution guidelines
Status icon key:
To do
Under construction
New
Updated
Constant change
None - stable page
Smaller versions of status icons for inline text:
Copyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.