r59 - 2023-10-30 - 12:30:10 - VaughnRokoszYou are here: TWiki >  Deployment Web > DeploymentMonitoring

deploy_50.png Monitoring

Section technical leaders: VaughnRokosz, TimFeeney, MichaelRowe


An active deployment of the Engineering Lifecycle Management (ELM) solution is always changing. For example, the number of active users increases as people start their work day, and decreases during off-hours. Over longer time frames, the deployment grows in size as artifacts are created. And usage patterns may change as new ELM capabilities are adopted or as end users tackle new business problems.


Of course, changes in usage patterns or deployment size can require adjustments to your deployment topology. IBM recommends adopting a proactive approach based on a “Measure-analyze-act” control loop.

You measure key health metrics, analyze them to identify current (or potential) problems, and then act on your analysis to address the problems.

Measure

Start by identifying the key indicators of deployment health, and then measure the values of those metrics periodically. Keep a historical record of the measurements so you can see changes over time. IBM strongly recommends that you use a commercial tool for Application Performance Monitoring (APM) to do this.

Basic health indicators

You can get a good initial assessment of deployment health by monitoring usage metrics at the operating system level. This includes:

  • % CPU utilization
  • % CPU time spent waiting for disk
  • % disk busy
  • Free disk space

Collect this information for all servers that are part of the deployment (including proxy servers and database servers).

ELM health indicators

ELM applications surface health indicators through an implementation of J2EE industry standard managed beans, which are defined as part of the Java Management Extensions (JMX) specification. Managed beans provide a defined, well understood way of getting information about what ELM applications are doing.

Start by reviewing the introductory material here:

These articles introduce a set of starter metrics that can provide early indications of problems:

  • Active services and server activity summaries (including concurrent user load)
  • Resource usage (JDBC and RDB mediator pools)
  • Database metrics
  • Java Virtual Machine (JVM) metrics (including memory usage)
  • ELM diagnostic data
  • Resource-intensive scenarios

Documentation for additional MBeans can be found here:

You can also discover available MBeans (and inspect their attributes) from a running ELM Java process.

  • JConsole is a graphical tool that lists available MBeans in a navigation tree.
  • The repodebug UI that is available for most ELM applications includes an MBean browser that lets you search for MBeans and then inspect their values.

Other health indicators

Analyze

The “Analyze” part of the “Measure-analyze-act” loop involves looking at the data collected during measurement to identify current (or potential) problems. The simplest form of analysis is to assign a threshold to each metric. There is a potential issue if the metric routinely exceeds the threshold value. For example, a server may be overloaded if the percentage of used CPU exceeds 90%. You can also look at patterns over time. For example, you might only flag a CPU utilization issue if 7 out of the last 10 measurements exceeded 90%. You can also derive rates of change from your historical data. For example, you might see that used disk space is growing at a rate that means you’ll run out in 30 days. That might be more useful to you than waiting for used disk space to reach 95%. There will often be a correlation between metrics. For example, the response time of a service can be impacted by the CPU usage on a server, or by the number of transactions processed per second.

The role of APM tools

An APM tool can greatly simplify the analysis process. IBM does not have a specific recommendation for APM tools, although some customers have successfully deployed Splunk. Other options include IBM Instana Observability and Prometheus.

An APM tool should be able to do the following things:

  • Read values from JMX MBeans. Since most ELM performance data is made available through MBeans, this is a must-have feature.
  • Collect performance data from database servers
  • Collect performance data from proxy servers
  • Collect operating system metrics from physical or virtual servers
  • Display metrics in dashboards
  • Configure alerts on metrics and provide notifications when alerts are triggered

Act

The final step in the Measure-Analyze-Act loop is to take action based on your analysis of the measurements. Some of the actions that you might take after analysis include:

  • Update the hardware on a server. Add more CPUs, memory or disk space.
  • Tune the JVM parameters (usually heap settings)
  • Adjust configuration parameters for the ELM applications (or for other software in the deployment like database or proxy servers)
  • Deploy a new ELM application server and create projects on that server to balance workload
  • Set up an entirely new deployment (e.g. JTS + applications)
  • Restart an application or server
  • Manually terminate long-running database queries
  • Collect more detailed diagnostics or “must gather” data to drill deeper into problem areas
  • Cluster a portion of the deployment. Add or remove cluster nodes based on changes in load.
  • Open a Salesforce case and create APARs if problems are due to defects
  • Upgrade to a new ifix or a new major release
  • Adopt new product features

After you take action, you start the Measure-Analyze-Act cycle again.

Example use cases

References

Topic attachments
I Attachment Action Size Date Who Comment
Pngpng MAA.png manage 5.8 K 2023-10-28 - 16:01 VaughnRokosz  
Edit | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r59 < r58 < r57 < r56 < r55 | More topic actions
 
This site is powered by the TWiki collaboration platformCopyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.