r3 - 2015-04-15 - 14:16:27 - GeraldMitchellYou are here: TWiki >  Deployment Web > DeploymentMonitoring > CLMServerMonitoring > CLMServerMonitoringJMXConnectionTroubleshooting

CLM Server Monitoring Connection Troubleshooting todo.png

Authors: GeraldMitchell
Build basis: CLM 4.0.6, CSM Agent 5.0.1


This section contains information about how the connections operate, and how to troubleshoot connection problems.

Basic Information

SOAP connector method to access the RMI-IIOP connector

JMX connector protocol definitions


Simple Object Access Protocol (SOAP)


Remote Method Invocation (RMI)


JMX Remote application programming interface (JSR 160) Remote Method Invocation (RMI)


Inter-Process Communications (IPC)

The following ports are used for making Java™™ Management Extensions (JMX) connections with the server: RMI, also known as the ORB bootstrap, port is designed to improve performance and communication with the server. The RMI connection is JSR 160 RMI compliant. The default setting of the RMI port is 2809. The SOAP connector port is more firewall compatible. The default setting of the SOAP port is 8880 The InterProcess Connector (IPC) port is a more stable and robust connection between systems on a local server. The default setting of the IPC port is 9633

If you want to connect to a WebSphere® Application Server on another server, you need to use the SOAP connector port to cross the firewall that is typically present between the application servers.

Diagnosis using JConsole

Assuring that the JMX capability is active in CLM

Assure than a connection can be made from the CLM server machine via the JMX connection options chosen. From the CLM server, follow the instructions for connecting to the CLM Server using JConsole.

Assuring that the JMX connection can be made from the CSM server

Assure that a connection can be made from the CSM agent server to the CLM server, if on different hosted operating systems, via the JMX connection options chosen. From the CSM Agent server, follow the instructions for connecting to the CLM Server using JConsole.

Diagnosis using WebSphere capabilities

Reference Points

Development using JMX Remote API and WebSphere

Basics for WAS

WAS Must-Gather documentation specifically of interest for JMX: [[http://www-01.ibm.com/support/docview.wss?uid=swg21196218&context=SSCPPRM#show-hide][MustGather for SOAP issues]. Use the Manual steps for gathering specific data for JMX

  • Do the following to test and collect the correct information for understanding the calls made by the client program to the Application Server JMX APIs:
  • Enable tracing on the client JMX API program.
  • Copy the TraceSettings.properties file found in the WebSphere Application Server install_root/properties directory into the JMX API client classpath.
  • Modify the TraceSettings.properties file by changing the last line.
    • From: com.ibm.ejs.ras.*=all=enabled
    • To: *=all=enabled
  • Also verify the location of the traceFileName file is valid.
  • Add -DtraceSettingsFile=TraceSettings.properties to the JMX API client Java™ execution line.
    • Example: java -DtraceSettingsFile=TraceSettings.properties -classpath %WAS_install%\lib\admin.jar;%WAS_install%\lib\wasjmx.jar;. AdminClientExample
  • Enable Admin tracing on the server the client program connects to as well as any other servers that will be involved, using the Administrative Console.
  • Expand Troubleshooting and select Logs and Trace.
  • Select the problem server and select Change Log Detail Levels.
  • Set the trace entry to the following: com.ibm.websphere.management.*=all:com.ibm.ws.management.*=all
  • Make sure the Diagnostic Trace Service for this server is setup to log the information to a file.
  • Click OK and save or synchronize changes.
  • Stop the server(s) and clear the server(s) logs (ffdc too).
  • Recreate the problem
  • (Run the collector tool on the problem profile(s).)
  • Collect the following files for support:
    • Collector output jar
    • JMX API client trace
    • Simple client or snippet of the problem code

And the ORB trace ORB Must Gather specifically WebSphere Application Server V6.1, V7.0, V8.0, and V8.5 ORB trace instructions

  • From the Administrative Console, select Servers > Application Servers > server_name > Change Log Details Levels.
  • Remove any previous entries in the text field type the following:*=info:ORBRas=all
  • Apply and save changes.
  • Select Servers > Application Servers > server_name > Diagnostic Trace Service.
  • Change the Maximum Number of Historical Files to 10 and maximum file size to 50mb
  • Apply and save changes.
  • Enabling Comm Trace:
    • Application Server: From the Administrative Console, Select Servers > Application Servers > server_name > Container Services > ORB Service. Select the Orb Tracing check box to enable Comm Tracing.
    • Node Agent: From the Administrative Console, select System Administration >Node Agent > nodeagent > ORB Service. Select the Orb Tracing check box to enable Comm Tracing.
    • Deployment Manager: From the Administrative Console, Select System Administration > Deployment Manager > ORB Service. Select the Orb Tracing check box to enable Comm Tracing.
  • Apply and save changes.
  • Restart the sever and recreate the problem
  • Collect the following file: profile_root/logs/server_name/trace.log
  • Notes: If your trace rolls over (timestamp appended), send all files with the trace prefix.
  • Refer to Steps for enabling traces (ORB) at runtime
    • Stand Alone Java Client ORB Trace Instructions
      • Start the client program with the "-D" arguments to specify the trace settings -Dcom.ibm.CORBA.Debug=true -Dcom.ibm.CORBA.CommTrace=true -Dcom.ibm.CORBA.Debug.Output=client.log
      • The ORB trace output is captured in the path pointed by com.ibm.CORBA.Debug.Output.
      • If the com.ibm.CORBA.Debug.Output parameter is not specified, the ORB trace output is captured in a unique trace file named orbtrc..txt in the current directory of execution.
      • Collect the following file: orbtrc..txt
    • J2EE Client Trace Instructions
      • Start the launchClient script with the following arguments to enable the trace: install_root/bin/launchClient.sh -JVMOptions="-Dcom.ibm.CORBA.Debug=true -Dcom.ibm.CORBA.CommTrace=true" -CCtrace=ORBRas=all -CCtracefile=orbtrace.txt -CCtraceMode=basic
      • Note: For Windows installations, use launchClient.bat instead of launchClient.sh
      • The ORB trace output is captured in a unique trace file named orbtrace.txt in the current directory of execution.
      • Collect the following file:orbtrace.txt
    • How to check IBM® Java™ ORB build version in WebSphere® Application Server.
      • /java/bin/java -Xbootclasspath/p:\java\jre\lib\ext\ibmorb.jar com.ibm.rmi.util.Version

If you have a IHS or Apache web server, as a proxy or otherwise : Must Gather for IBM HTTP Server

Access using the administrative console page for JMX connectors

  • information : WAS8.0 JMX Connector Definitions
  • click Servers > Server Types > WebSphere application servers > server_name > Administration > Administration services > JMX Connectors
    • If Yes is specified, the JMX connector is enabled. All JMX connectors are enabled by default. To disable a JMX connector, select the JMX connector and click Disable. Save the configuration changes if prompted.

WAS 8.5.5 Web services client runtime troubleshooting tips

WAS 8.0 JMX Connector properties

WAS8.0 JMX Connector Definitions

SOAP and IPC connector file definition for soap.client.props and ipc.client.props SSL client file definition for ssl.clent.props

WAS 8.0 interface for Java dumps, heaps, and system dumps


Technote 1254706 Setting up a trace in WebSphere Application Server

basic and HPEL logging

N1019742 tracing SOAP connectivity issues excerpt:

  1. Use the -trace option on the addNode/removeNode Qshell commands i.e. app_server_root/bin/addNode -conntype SOAP -profileName -trace
  2. Specify the following diagnostic trace string for the Deployment Manager you are attempting to connect to: *=info:com.ibm.ws.management.*=all:com.ibm.websphere.management.application.*=all:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:TCPChannel=all:GenericBNF=all
For detailed instructions on how to set up diagnostic tracing in WebSphere Application Server, please refer to the following URL: http://www-01.ibm.com/support/docview.wss?uid=swg21254706

3) TCP/IP traces from the both the DMGR and Application Server perspectives. Use the TRCCNN command to gather the TCP/IP traces.

Command from the Application Server (who is connecting to the DMGR) perspective: TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP () (SOAPPORT)) TRCCNN SET(*OFF) TRCTBL(IBM)


where SOAPPORT is the SOAP port of the DMGR you are attempting to connect to.

reference of JVM Custom Properties

Basics of WAS Monitoring

Introduction to WAS monitoring

Known issues and Workarounds

In GA versions of CSM Monitoring Agent 5.0.1 and 5.0.2, there are a few issues in the connections page setup.

CSM Agent user interface behavior and errors

I thought my Connections page errors should be resolved after making changes but I still see the same error

  • There is a cache of the connections information for 5 minutes, causing some error messages to be erroneously shown, active or inactive annotations incorrect, and other User Interface problems with information reliability. As a workaround, please do a browser level refresh from the page to restart the page logic and reset the information to assure of correct or incorrect information being displayed.

A Loading... message and CRJAZ00241 An Error occurred retrieving from location "virtual/jmx/domains ... is seen in the JMX connections page.

  • Reason: when using an Apache-based proxy such as IBM HTTP Server, there are possible issues in the "/" slash characters as %2F in URI locations when transmitted through a proxy where the slash is not treated as part of the URI as well as possible "@" and "./" combinations in the URL. The IHS may be interpreting the information as paths, differentiating from paths, or encoding information depending on the IHS configuration.
  • It is recommended to not access the CSM Agent through IHS if possible, as the IHS is analyzing and modifying the URLs in these scenarios. If possible, set the bypass the IHS to access the CSM Agent user interface (/csm). In typical scenarios the application server port (such as 9444) should be used in the /csm public URI, and the administrator should access CSM through the application server port.
  • If the IHS is absolutely required on the CSM Agent, changes to the IHS default settings must be made such that the IHS does not modify the request or response from the CSM Agent User Interface in the browser and the CSM Agent Servlet on the CSM application server. This is IHS configuration dependent. AllowPathInfo and AllowEncodedSlashes settings may need to be added to both the top level httpd.conf and the sections. In some cases mod-proxy and mod-rewrite may be necessary and a rewrite pattern employed.

Related topics: Deployment web home, Deployment web home

External links:

Additional contributors: TWikiUser, TWikiUser

Edit | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r6 < r5 < r4 < r3 < r2 | More topic actions...
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Ideas, requests, problems regarding the Deployment wiki? Create a new task in the RTC Deployment wiki project