CLM Server Monitoring Connection Troubleshooting (DEPRECATED: CLM Server Monitoring is no longer supported and was dropped in version 6.0.3.)

Authors: GeraldMitchell
Build basis: CLM 4.0.6, CSM Agent 5.0.1

UNDER CONSTRUCTION

This section contains information about how the connections operate, and how to troubleshoot connection problems.

Basic Information

SOAP connector method to access the RMI-IIOP connector

JMX connector protocol definitions

SOAP

Simple Object Access Protocol (SOAP)

RMI

Remote Method Invocation (RMI)

JSR160RMI

JMX Remote application programming interface (JSR 160) Remote Method Invocation (RMI)

IPC

Inter-Process Communications (IPC)

The following ports are used for making Java™™ Management Extensions (JMX) connections with the server: RMI, also known as the ORB bootstrap, port is designed to improve performance and communication with the server. The RMI connection is JSR 160 RMI compliant. The default setting of the RMI port is 2809. The SOAP connector port is more firewall compatible. The default setting of the SOAP port is 8880 The InterProcess Connector (IPC) port is a more stable and robust connection between systems on a local server. The default setting of the IPC port is 9633

If you want to connect to a WebSphere® Application Server on another server, you need to use the SOAP connector port to cross the firewall that is typically present between the application servers.

Diagnosis using JConsole

Assuring that the JMX capability is active in CLM

Assure than a connection can be made from the CLM server machine via the JMX connection options chosen. From the CLM server, follow the instructions for connecting to the CLM Server using JConsole.

Assuring that the JMX connection can be made from the CSM server

Assure that a connection can be made from the CSM agent server to the CLM server, if on different hosted operating systems, via the JMX connection options chosen. From the CSM Agent server, follow the instructions for connecting to the CLM Server using JConsole.

Diagnosis using WebSphere capabilities

Reference Points

Development using JMX Remote API and WebSphere

TroubleShooting: JMX API for WebSphere Application Server USE THIS

Check for wasadmin connection capability

Normally we would use JConsole and the script to check the CLM hosted WebSphere Security settings and transfer the certificate. For some platforms and configurations, JConsole is not an option.

We can try to see if the wsadmin, which is built on the same substrates of JMX and AdminClient code as the CSM Agent, can connect. This should be using the same security and SOAP->JMX path as the CSM Agent is taking in WebSphere and so is a good confirmation.

Use the wasdmin command remotely from the CSM Agent server to the CLM server see if the ports and security are good. This command supports all the same connection types (RMI, SOAP, etc.)

• /opt/IBM/WebSphere/AppServer/bin/wsadmin.sh -conntype SOAP -host CLM_APPLICATION_SERVER_ADDRESS -port _CLM_APPLICATION_SERVER_PORT_, and you will be prompted for user and password.

• /opt/IBM/WebSphere/AppServer/bin/wsadmin.sh -conntype SOAP -host CLM_APPLICATION_SERVER_ADDRESS -port CLM_APPLICATION_SERVER_PORT -username WASADMIN -password WASADMINPASSWORD_ NOTE: if you type the password into the command, using -username _WASADMIN -password WASADMINPASSWORD you won't get a prompt for username and password, but the username and password are in the service name invoked and so show in the system ps and logs etc which may be a consideration for you security)

You may see the following message: "A retry of the request may need to occur if the socket times out while waiting for a prompt response. If the retry is required, note that the prompt will not be redisplayed if is entered, which indicates the signer has already been added to the trust store."

• On systems without a GUI, the popup process should wait over the 30 seconds and then fail, at which point you should get a prompt for userid and password.

If you issue this on Windows/Linux with XWindows without the username and password in the command, you may get a Java popup. Wait for it to time out (30 seconds) or fill in the information.

• On systems without a GUI, the popup process should wait over the 30 seconds and then fail, at which point you should get a prompt for userid and password.

Use the WASADMIN user name and password to login. You should see a prompt "wsadmin", at which point type "quit" and press enter/return.

In CSM, after this is completed, make sure to create a NEW connection with the same CLM_APPLICATION_SERVER_ADDRESS and CLM_APPLICATION_SERVER_PORT using WebSphere, and SOAP and the same WASADMIN user name and password for the secure connection. Do NOT attempt to reuse a previous connection that did not connect.

If you look at wsadmin in the WebSphere 8.5 documentation on KnowledgeCenter you can see all of the commands and information.

Basics for WAS

WAS Must-Gather documentation specifically of interest for JMX: [[http://www-01.ibm.com/support/docview.wss?uid=swg21196218&context=SSCPPRM#show-hide][MustGather for SOAP issues]. Use the Manual steps for gathering specific data for JMX

• Do the following to test and collect the correct information for understanding the calls made by the client program to the Application Server JMX APIs:
• Enable tracing on the client JMX API program.
• Copy the TraceSettings.properties file found in the WebSphere Application Server install_root/properties directory into the JMX API client classpath.
• Modify the TraceSettings.properties file by changing the last line.
  • From: com.ibm.ejs.ras.*=all=enabled
  • To: *=all=enabled
• Also verify the location of the traceFileName file is valid.
• Add -DtraceSettingsFile=TraceSettings.properties to the JMX API client Java™ execution line.
  • Example: java -DtraceSettingsFile=TraceSettings.properties -classpath %WAS_install%\lib\admin.jar;%WAS_install%\lib\wasjmx.jar;. AdminClientExample
• Enable Admin tracing on the server the client program connects to as well as any other servers that will be involved, using the Administrative Console.
• Expand Troubleshooting and select Logs and Trace.
• Select the problem server and select Change Log Detail Levels.
• Set the trace entry to the following: com.ibm.websphere.management.*=all:com.ibm.ws.management.*=all
• Make sure the Diagnostic Trace Service for this server is setup to log the information to a file.
• Click OK and save or synchronize changes.
• Stop the server(s) and clear the server(s) logs (ffdc too).
• Recreate the problem
• (Run the collector tool on the problem profile(s).)
• Collect the following files for support:
  • Collector output jar
  • JMX API client trace
  • Simple client or snippet of the problem code

And the ORB trace ORB Must Gather specifically WebSphere Application Server V6.1, V7.0, V8.0, and V8.5 ORB trace instructions

• From the Administrative Console, select Servers > Application Servers > server_name > Change Log Details Levels.
• Remove any previous entries in the text field type the following:*=info:ORBRas=all
• Apply and save changes.
• Select Servers > Application Servers > server_name > Diagnostic Trace Service.
• Change the Maximum Number of Historical Files to 10 and maximum file size to 50mb
• Apply and save changes.
• Enabling Comm Trace:
  • Application Server: From the Administrative Console, Select Servers > Application Servers > server_name > Container Services > ORB Service. Select the Orb Tracing check box to enable Comm Tracing.
  • Node Agent: From the Administrative Console, select System Administration >Node Agent > nodeagent > ORB Service. Select the Orb Tracing check box to enable Comm Tracing.
  • Deployment Manager: From the Administrative Console, Select System Administration > Deployment Manager > ORB Service. Select the Orb Tracing check box to enable Comm Tracing.
• Apply and save changes.
• Restart the sever and recreate the problem
• Collect the following file: profile_root/logs/server_name/trace.log
• Notes: If your trace rolls over (timestamp appended), send all files with the trace prefix.
• Refer to Steps for enabling traces (ORB) at runtime
  • Stand Alone Java Client ORB Trace Instructions
    • Start the client program with the "-D" arguments to specify the trace settings -Dcom.ibm.CORBA.Debug=true -Dcom.ibm.CORBA.CommTrace=true -Dcom.ibm.CORBA.Debug.Output=client.log
    • The ORB trace output is captured in the path pointed by com.ibm.CORBA.Debug.Output.
    • If the com.ibm.CORBA.Debug.Output parameter is not specified, the ORB trace output is captured in a unique trace file named orbtrc..txt in the current directory of execution.
    • Collect the following file: orbtrc..txt
  • J2EE Client Trace Instructions
    • Start the launchClient script with the following arguments to enable the trace: install_root/bin/launchClient.sh -JVMOptions="-Dcom.ibm.CORBA.Debug=true -Dcom.ibm.CORBA.CommTrace=true" -CCtrace=ORBRas=all -CCtracefile=orbtrace.txt -CCtraceMode=basic
    • Note: For Windows installations, use launchClient.bat instead of launchClient.sh
    • The ORB trace output is captured in a unique trace file named orbtrace.txt in the current directory of execution.
    • Collect the following file:orbtrace.txt
  • How to check IBM® Java™ ORB build version in WebSphere® Application Server.
    • /java/bin/java -Xbootclasspath/p:\java\jre\lib\ext\ibmorb.jar com.ibm.rmi.util.Version

If you have a IHS or Apache web server, as a proxy or otherwise : Must Gather for IBM HTTP Server

Access using the administrative console page for JMX connectors

• information : WAS8.0 JMX Connector Definitions
• click Servers > Server Types > WebSphere application servers > server_name > Administration > Administration services > JMX Connectors
  • If Yes is specified, the JMX connector is enabled. All JMX connectors are enabled by default. To disable a JMX connector, select the JMX connector and click Disable. Save the configuration changes if prompted.

WAS 8.5.5 Web services client runtime troubleshooting tips

WAS 8.0 JMX Connector properties

WAS8.0 JMX Connector Definitions

SOAP and IPC connector file definition for soap.client.props and ipc.client.props SSL client file definition for ssl.clent.props

WAS 8.0 interface for Java dumps, heaps, and system dumps

[[http://www.ibm.com/developerworks/library/co-websphere-access-feature/][]]

Technote 1254706 Setting up a trace in WebSphere Application Server

basic and HPEL logging

N1019742 tracing SOAP connectivity issues excerpt:

1. Use the -trace option on the addNode/removeNode Qshell commands i.e. app_server_root/bin/addNode -conntype SOAP -profileName -trace
2. Specify the following diagnostic trace string for the Deployment Manager you are attempting to connect to: *=info:com.ibm.ws.management.*=all:com.ibm.websphere.management.application.*=all:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:TCPChannel=all:GenericBNF=all

For detailed instructions on how to set up diagnostic tracing in WebSphere Application Server, please refer to the following URL: http://www-01.ibm.com/support/docview.wss?uid=swg21254706

3) TCP/IP traces from the both the DMGR and Application Server perspectives. Use the TRCCNN command to gather the TCP/IP traces.

Command from the Application Server (who is connecting to the DMGR) perspective: TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP () (SOAPPORT)) TRCCNN SET(*OFF) TRCTBL(IBM)

Command from the DMGR perspective: TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP (SOAPPORT) ()) TRCCNN SET(*OFF) TRCTBL(IBM)

where SOAPPORT is the SOAP port of the DMGR you are attempting to connect to.

reference of JVM Custom Properties

Basics of WAS Monitoring

Introduction to WAS monitoring

Known issues and Workarounds

In GA versions of CSM Monitoring Agent 5.0.1 and 5.0.2, there are a few issues in the connections page setup.

CSM Agent user interface behavior and errors

I thought my Connections page errors should be resolved after making changes but I still see the same error

• There is a cache of the connections information for 5 minutes, causing some error messages to be erroneously shown, active or inactive annotations incorrect, and other User Interface problems with information reliability. As a workaround, please do a browser level refresh from the page to restart the page logic and reset the information to assure of correct or incorrect information being displayed.

A Loading... message and CRJAZ00241 An Error occurred retrieving from location "virtual/jmx/domains ... is seen in the JMX connections page.

One possibility on CSM Agent 5.0.1 and 5.0.2: This may be seen Apache-based proxy such as IBM HTTP Server, there are possible issues in the "/" slash characters as %2F in URI locations when transmitted through a proxy where the slash is not treated as part of the URI as well as possible "@" and "./" combinations in the URL. The IHS may be interpreting the information as paths, differentiating from paths, or encoding information depending on the IHS configuration.

• It is recommended to not access the CSM Agent through IHS if possible, as the IHS is analyzing and modifying the URLs in these scenarios. If possible, set the bypass the IHS to access the CSM Agent user interface (/csm). In typical scenarios the application server port (such as 9444) should be used in the CSM Agent public URI, and the administrator should access CSM through the application server port.
• If the IHS is absolutely required on the CSM Agent, changes to the IHS default settings must be made such that the IHS does not modify the request or response from the CSM Agent User Interface in the browser and the CSM Agent Servlet on the CSM application server. This is IHS configuration dependent. AllowPathInfo and AllowEncodedSlashes settings may need to be added to both the top level httpd.conf and the sections. In some cases mod-proxy and mod-rewrite may be necessary and a rewrite pattern employed.

Related topics: Deployment web home, Deployment web home

External links:

• IBM

Additional contributors: TWikiUser, TWikiUser