EditAttachPrintable
r2 - 2015-03-21 - 20:31:44 - GeraldMitchellYou are here: TWiki >  Deployment Web > DeploymentMonitoring > CLMServerMonitoring > CLMServerMonitoringJMXConnectionTroubleshooting

CLM Server Monitoring Connection Troubleshooting todo.png

Authors: GeraldMitchell
Build basis: CLM 4.0.6, CSM Agent 5.0.1

UNDER CONSTRUCTION

This section contains information about how the connections operate, and how to troubleshoot connection problems.

Basic Information

SOAP connector method to access the RMI-IIOP connector

JMX connector protocol definitions

SOAP

Simple Object Access Protocol (SOAP)

RMI

Remote Method Invocation (RMI)

JSR160RMI

JMX Remote application programming interface (JSR 160) Remote Method Invocation (RMI)

IPC

Inter-Process Communications (IPC)

The following ports are used for making Java™™ Management Extensions (JMX) connections with the server: RMI, also known as the ORB bootstrap, port is designed to improve performance and communication with the server. The RMI connection is JSR 160 RMI compliant. The default setting of the RMI port is 2809. The SOAP connector port is more firewall compatible. The default setting of the SOAP port is 8880 The InterProcess Connector (IPC) port is a more stable and robust connection between systems on a local server. The default setting of the IPC port is 9633

If you want to connect to a WebSphere® Application Server on another server, you need to use the SOAP connector port to cross the firewall that is typically present between the application servers.

Diagnosis using JConsole

Assuring that the JMX capability is active in CLM

Assure than a connection can be made from the CLM server machine via the JMX connection options chosen. From the CLM server, follow the instructions for connecting to the CLM Server using JConsole.

Assuring that the JMX connection can be made from the CSM server

Assure that a connection can be made from the CSM agent server to the CLM server, if on different hosted operating systems, via the JMX connection options chosen. From the CSM Agent server, follow the instructions for connecting to the CLM Server using JConsole.

Diagnosis using WebSphere capabilities

Reference Points

Development using JMX Remote API and WebSphere

Basics for WAS

WAS Must-Gather documentation specifically of interest for JMX: [[http://www-01.ibm.com/support/docview.wss?uid=swg21196218&context=SSCPPRM#show-hide][MustGather for SOAP issues]. Use the Manual steps for gathering specific data for JMX

  • Do the following to test and collect the correct information for understanding the calls made by the client program to the Application Server JMX APIs:
  • Enable tracing on the client JMX API program.
  • Copy the TraceSettings.properties file found in the WebSphere Application Server install_root/properties directory into the JMX API client classpath.
  • Modify the TraceSettings.properties file by changing the last line.
    • From: com.ibm.ejs.ras.*=all=enabled
    • To: *=all=enabled
  • Also verify the location of the traceFileName file is valid.
  • Add -DtraceSettingsFile=TraceSettings.properties to the JMX API client Java™ execution line.
    • Example: java -DtraceSettingsFile=TraceSettings.properties -classpath %WAS_install%\lib\admin.jar;%WAS_install%\lib\wasjmx.jar;. AdminClientExample
  • Enable Admin tracing on the server the client program connects to as well as any other servers that will be involved, using the Administrative Console.
  • Expand Troubleshooting and select Logs and Trace.
  • Select the problem server and select Change Log Detail Levels.
  • Set the trace entry to the following: com.ibm.websphere.management.*=all:com.ibm.ws.management.*=all
  • Make sure the Diagnostic Trace Service for this server is setup to log the information to a file.
  • Click OK and save or synchronize changes.
  • Stop the server(s) and clear the server(s) logs (ffdc too).
  • Recreate the problem
  • (Run the collector tool on the problem profile(s).)
  • Collect the following files for support:
    • Collector output jar
    • JMX API client trace
    • Simple client or snippet of the problem code

And the ORB trace ORB Must Gather specifically WebSphere Application Server V6.1, V7.0, V8.0, and V8.5 ORB trace instructions

  • From the Administrative Console, select Servers > Application Servers > server_name > Change Log Details Levels.
  • Remove any previous entries in the text field type the following:*=info:ORBRas=all
  • Apply and save changes.
  • Select Servers > Application Servers > server_name > Diagnostic Trace Service.
  • Change the Maximum Number of Historical Files to 10 and maximum file size to 50mb
  • Apply and save changes.
  • Enabling Comm Trace:
    • Application Server: From the Administrative Console, Select Servers > Application Servers > server_name > Container Services > ORB Service. Select the Orb Tracing check box to enable Comm Tracing.
    • Node Agent: From the Administrative Console, select System Administration >Node Agent > nodeagent > ORB Service. Select the Orb Tracing check box to enable Comm Tracing.
    • Deployment Manager: From the Administrative Console, Select System Administration > Deployment Manager > ORB Service. Select the Orb Tracing check box to enable Comm Tracing.
  • Apply and save changes.
  • Restart the sever and recreate the problem
  • Collect the following file: profile_root/logs/server_name/trace.log
  • Notes: If your trace rolls over (timestamp appended), send all files with the trace prefix.
  • Refer to Steps for enabling traces (ORB) at runtime
    • Stand Alone Java Client ORB Trace Instructions
      • Start the client program with the "-D" arguments to specify the trace settings -Dcom.ibm.CORBA.Debug=true -Dcom.ibm.CORBA.CommTrace=true -Dcom.ibm.CORBA.Debug.Output=client.log
      • The ORB trace output is captured in the path pointed by com.ibm.CORBA.Debug.Output.
      • If the com.ibm.CORBA.Debug.Output parameter is not specified, the ORB trace output is captured in a unique trace file named orbtrc..txt in the current directory of execution.
      • Collect the following file: orbtrc..txt
    • J2EE Client Trace Instructions
      • Start the launchClient script with the following arguments to enable the trace: install_root/bin/launchClient.sh -JVMOptions="-Dcom.ibm.CORBA.Debug=true -Dcom.ibm.CORBA.CommTrace=true" -CCtrace=ORBRas=all -CCtracefile=orbtrace.txt -CCtraceMode=basic
      • Note: For Windows installations, use launchClient.bat instead of launchClient.sh
      • The ORB trace output is captured in a unique trace file named orbtrace.txt in the current directory of execution.
      • Collect the following file:orbtrace.txt
    • How to check IBM® Java™ ORB build version in WebSphere® Application Server.
      • /java/bin/java -Xbootclasspath/p:\java\jre\lib\ext\ibmorb.jar com.ibm.rmi.util.Version

If you have a IHS or Apache web server, as a proxy or otherwise : Must Gather for IBM HTTP Server

Access using the administrative console page for JMX connectors

  • information : WAS8.0 JMX Connector Definitions
  • click Servers > Server Types > WebSphere application servers > server_name > Administration > Administration services > JMX Connectors
    • If Yes is specified, the JMX connector is enabled. All JMX connectors are enabled by default. To disable a JMX connector, select the JMX connector and click Disable. Save the configuration changes if prompted.

WAS 8.5.5 Web services client runtime troubleshooting tips

WAS 8.0 JMX Connector properties

WAS8.0 JMX Connector Definitions

SOAP and IPC connector file definition for soap.client.props and ipc.client.props SSL client file definition for ssl.clent.props

WAS 8.0 interface for Java dumps, heaps, and system dumps

[[http://www.ibm.com/developerworks/library/co-websphere-access-feature/][]]

Technote 1254706 Setting up a trace in WebSphere Application Server

basic and HPEL logging

N1019742 tracing SOAP connectivity issues excerpt:

  1. Use the -trace option on the addNode/removeNode Qshell commands i.e. app_server_root/bin/addNode -conntype SOAP -profileName -trace
  2. Specify the following diagnostic trace string for the Deployment Manager you are attempting to connect to: *=info:com.ibm.ws.management.*=all:com.ibm.websphere.management.application.*=all:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:TCPChannel=all:GenericBNF=all
For detailed instructions on how to set up diagnostic tracing in WebSphere Application Server, please refer to the following URL: http://www-01.ibm.com/support/docview.wss?uid=swg21254706

3) TCP/IP traces from the both the DMGR and Application Server perspectives. Use the TRCCNN command to gather the TCP/IP traces.

Command from the Application Server (who is connecting to the DMGR) perspective: TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP () (SOAPPORT)) TRCCNN SET(*OFF) TRCTBL(IBM)

Command from the DMGR perspective: TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP (SOAPPORT) ()) TRCCNN SET(*OFF) TRCTBL(IBM)

where SOAPPORT is the SOAP port of the DMGR you are attempting to connect to.

reference of JVM Custom Properties

Basics of WAS Monitoring

Introduction to WAS monitoring

Known issues and Workarounds

In GA versions of CSM Monitoring Agent 5.0.1 and 5.0.2, there are a few issues in the connections page setup.

CSM Agent user interface behavior and errors

I thought my Connections page errors should be resolved after making changes but I still see the same error

  • There is a cache of the connections information for 5 minutes, causing some error messages to be erroneously shown, active or inactive annotations incorrect, and other User Interface problems with information reliability. As a workaround, please do a browser level refresh from the page to restart the page logic and reset the information to assure of correct or incorrect information being displayed. {Technote to be posted here when available. }

A Loading... message and CRJAZ00241 An Error occurred retrieving from location "virtual/jmx/domains ... is seen in the JMX connections page.

  • When using an Apache-based proxy such as IBM HTTP Server, there is a problem with "/" slash characters as %2F in URI locations when transmitted through a proxy where the slash is not treated as part of the URI. This issue currently requires a workaround completed by altering the proxy settings in httpd.conf to have entries for CLM applications and components ( including JTS and CSM Agent ) to contain "AllowEncodedSlashes ON" in order to pass the messages from the Browser to the CSM Agent as intended. {Technote to be posted here when available. }

Related topics: Deployment web home, Deployment web home

External links:

Additional contributors: TWikiUser, TWikiUser

Edit | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r6 | r4 < r3 < r2 < r1 | More topic actions...
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Ideas, requests, problems regarding the Deployment wiki? Create a new task in the RTC Deployment wiki project