It's all about the answers!

Ask a question

Rational Application Suddenly Stopped


1
1
Irell Carloe Santos (241215) | asked Jul 01 '15, 9:41 p.m.
edited Jul 01 '15, 9:56 p.m.
Hi,

Rational Application suddenly stopped and we have to restart the whole machine for it to run correctly again.
Please see below logs on what caused this downtime and advise on a possible preventive action. Thank you very much.

[7/1/15 17:31:26:274 SGT] 00000042 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\ffdc\server1_73c273c2_15.07.01_17.31.26.2741905349994276584417.txt com.ibm.ws.ssl.channel.impl.SSLReadServiceContext 192
[7/1/15 17:32:54:741 SGT] 000000a6 AdminHelper   A   ADMN1020I: An attempt is made to stop the server1 server. (User ID = defaultWIMFileBasedRealm/wasadm)
[7/1/15 17:32:54:866 SGT] 00000004 TCPChannel    I   TCPC0002I: TCP Channel TCP_4 has stopped listening on host *  (IPv6) port 9443.
[7/1/15 17:32:54:881 SGT] 00000004 TCPChannel    I   TCPC0002I: TCP Channel TCP_2 has stopped listening on host *  (IPv6) port 9080.
[7/1/15 17:33:00:029 SGT] 00000004 ApplicationMg A   WSVR0217I: Stopping application: jts_war
[7/1/15 17:33:14:381 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [jts_war] [/jts] [equinoxbridgeservlet]: Destroy successful.
[7/1/15 17:33:14:444 SGT] 00000004 ApplicationMg A   WSVR0220I: Application stopped: jts_war
[7/1/15 17:33:14:444 SGT] 00000004 ApplicationMg A   WSVR0217I: Stopping application: rm_war
[7/1/15 17:33:20:699 SGT] 00000041 webcontainer  E com.ibm.ws.webcontainer.WebContainer sendUnavailableException Servlet.has.become.temporarily.unavailable.for.service
[7/1/15 17:33:29:404 SGT] 000000be SystemOut     O 2015-07-01 17:33:29,404 RQM [qm: AsynchronousTaskRunner-0 @@ 17:33] ERROR com.ibm.team.repository                             - CRJAZ2388E The background process that synchronizes user data between servers could not be completed. The synchronizer will attempt the synchronization again.
CRJAZ1166I The stack trace hash is 1F633DBB0D636D4AB6F6D104B45444611AE23D61.
com.ibm.team.repository.common.TeamRepositoryException: CRJAZ2051E The Jazz Team Server could not be connected to at host: "cbcrt.cbcmain.cbc.chinabank.ph:9443".  This could be because the server has not been started or it is behind a firewall and not accessible.
    at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.doCommunicate(CrossServerHttpClient.java:191)
    at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.send(CrossServerHttpClient.java:447)
    at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.send(CrossServerHttpClient.java:336)
    at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClient.send(CrossServerHttpClient.java:460)
    at com.ibm.team.repository.service.compatibility.util.http.CrossServerHttpClientService.send(CrossServerHttpClientService.java:68)
    at sun.reflect.GeneratedMethodAccessor354.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
    at $Proxy1481.send(Unknown Source)
    at com.ibm.team.repository.service.compatibility.AbstractCompatibilityService.send(AbstractCompatibilityService.java:56)
    at com.ibm.team.repository.service.compatibility.contributor.internal.CompatContributorService.get(CompatContributorService.java:741)
    at com.ibm.team.repository.service.compatibility.contributor.internal.CompatContributorService.query(CompatContributorService.java:717)
    at com.ibm.team.repository.service.compatibility.contributor.internal.ServicesBasedContributorSynchronizerHelper.queryUrl(ServicesBasedContributorSynchronizerHelper.java:108)
    at com.ibm.team.repository.service.compatibility.contributor.internal.ContributorSynchronizer.runQuery(ContributorSynchronizer.java:712)
    at com.ibm.team.repository.service.compatibility.contributor.internal.ContributorSynchronizer.runQuery(ContributorSynchronizer.java:699)
    at com.ibm.team.repository.service.compatibility.contributor.internal.ContributorSynchronizer.doPerformSynchronization(ContributorSynchronizer.java:259)
    at com.ibm.team.repository.service.compatibility.contributor.internal.ContributorSynchronizer.performSynchronization(ContributorSynchronizer.java:196)
    at com.ibm.team.repository.service.compatibility.contributor.internal.CompatContributorService$2.run(CompatContributorService.java:470)
    at com.ibm.team.repository.service.internal.TeamServiceContext.runAsAdmin(TeamServiceContext.java:141)
    at com.ibm.team.repository.service.compatibility.contributor.internal.CompatContributorService.performSynchronization(CompatContributorService.java:466)
    at com.ibm.team.repository.service.compatibility.contributor.internal.CompatContributorService.performSynchronization(CompatContributorService.java:435)
    at sun.reflect.GeneratedMethodAccessor398.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
    at $Proxy1594.performSynchronization(Unknown Source)
    at com.ibm.team.repository.service.compatibility.contributor.internal.ContributorSynchronizingTask.runTask(ContributorSynchronizingTask.java:38)
    at com.ibm.team.repository.service.async.AbstractAutoScheduledTask.executeTask(AbstractAutoScheduledTask.java:92)
    at sun.reflect.GeneratedMethodAccessor142.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)
    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)
    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
    at $Proxy1596.executeTask(Unknown Source)
    at com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.runTask(AsynchronousTaskRunner.java:158)
    at com.ibm.team.repository.service.internal.scheduler.AsynchronousTaskRunner.run(AsynchronousTaskRunner.java:119)
    at java.lang.Thread.run(Thread.java:738)
[7/1/15 17:33:34:864 SGT] 000000bf AlarmThreadMo W   UTLS0008W: The return of alarm thread "Non-deferrable Alarm : 0" (00000004) to the alarm thread pool has been delayed for 40045 milliseconds. This may be preventing normal alarm function within the application server. The alarm listener stack trace is as follows:
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:222)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2036)
    at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1264)
    at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.awaitTermination(MoreExecutors.java:424)
    at com.ibm.rdm.fronting.server.concurrent.RRSConcurrency.shutdown(RRSConcurrency.java:338)
    at com.ibm.rdm.fronting.server.concurrent.RRSConcurrency.shutdownAll(RRSConcurrency.java:324)
    at com.ibm.rdm.fronting.server.core.internal.RRSActivator.stop(RRSActivator.java:200)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:843)
    at java.security.AccessController.doPrivileged(AccessController.java:277)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:836)
    at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:501)
    at org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(AbstractBundle.java:550)
    at org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Framework.java:1097)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.decFWSL(StartLevelManager.java:597)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:257)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.shutdown(StartLevelManager.java:215)
    at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.suspend(InternalSystemBundle.java:266)
    at org.eclipse.osgi.framework.internal.core.Framework.shutdown(Framework.java:690)
    at org.eclipse.osgi.framework.internal.core.Framework.close(Framework.java:588)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.shutdown(EclipseStarter.java:415)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.eclipse.equinox.servletbridge.FrameworkLauncher.stop(FrameworkLauncher.java:719)
    at org.eclipse.equinox.servletbridge.BridgeServlet.destroy(BridgeServlet.java:77)
    at com.ibm.team.repository.server.servletbridge.JazzServlet.destroy(JazzServlet.java:90)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.doDestroy(ServletWrapper.java:1436)
    at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.doDestroy(ServletWrapperImpl.java:386)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.destroy(ServletWrapper.java:1548)
    at com.ibm.ws.webcontainer.webapp.WebApp.destroy(WebApp.java:3123)
    at com.ibm.ws.webcontainer.webapp.WebAppImpl.destroy(WebAppImpl.java:805)
    at com.ibm.ws.container.AbstractContainer.destroy(AbstractContainer.java:74)
    at com.ibm.ws.webcontainer.webapp.WebGroup.destroy(WebGroup.java:203)
    at com.ibm.ws.webcontainer.webapp.WebGroup.removeWebApplication(WebGroup.java:241)
    at com.ibm.ws.webcontainer.VirtualHost.removeWebApplication(VirtualHost.java:309)
    at com.ibm.ws.webcontainer.VirtualHostImpl.removeWebApplication(VirtualHostImpl.java:208)
    at com.ibm.ws.webcontainer.WSWebContainer.removeWebApplication(WSWebContainer.java:803)
    at com.ibm.ws.webcontainer.component.WebContainerImpl.uninstall(WebContainerImpl.java:408)
    at com.ibm.ws.webcontainer.component.WebContainerImpl.stop(WebContainerImpl.java:679)
    at com.ibm.ws.runtime.component.ApplicationMgrImpl.stop(ApplicationMgrImpl.java:1150)
    at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStop(DeployedApplicationImpl.java:1324)
    at com.ibm.ws.runtime.component.DeployedModuleImpl.stop(DeployedModuleImpl.java:646)
    at com.ibm.ws.runtime.component.DeployedApplicationImpl.stop(DeployedApplicationImpl.java:1098)
    at com.ibm.ws.runtime.component.ApplicationMgrImpl.stopApplication(ApplicationMgrImpl.java:921)
    at com.ibm.ws.runtime.component.ApplicationMgrImpl.stop(ApplicationMgrImpl.java:874)
    at com.ibm.ws.runtime.component.ContainerHelper.stopComponent(ContainerHelper.java:475)
    at com.ibm.ws.runtime.component.ContainerHelper.stopComponents(ContainerHelper.java:459)
    at com.ibm.ws.runtime.component.ContainerImpl.stopComponents(ContainerImpl.java:650)
    at com.ibm.ws.runtime.component.ContainerImpl.stop(ContainerImpl.java:638)
    at com.ibm.ws.runtime.component.ApplicationServerImpl.stop(ApplicationServerImpl.java:257)
    at com.ibm.ws.runtime.component.ContainerHelper.stopComponent(ContainerHelper.java:475)
    at com.ibm.ws.runtime.component.ContainerHelper.stopComponents(ContainerHelper.java:459)
    at com.ibm.ws.runtime.component.ContainerImpl.stopComponents(ContainerImpl.java:650)
    at com.ibm.ws.runtime.component.ContainerImpl.stop(ContainerImpl.java:638)
    at com.ibm.ws.runtime.component.ServerImpl.stop(ServerImpl.java:615)
    at com.ibm.ws.runtime.component.ServerCollaborator$ShutdownHook$1.run(ServerCollaborator.java:738)
    at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5365)
    at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5453)
    at com.ibm.ws.runtime.component.ServerCollaborator$ShutdownHook.run(ServerCollaborator.java:728)
    at com.ibm.ws.runtime.component.ServerCollaborator$StopAction.alarm(ServerCollaborator.java:692)
    at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:133)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646).
[7/1/15 17:33:45:753 SGT] 00000004 servlet       I   Servlet Destroyed
[7/1/15 17:33:46:002 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [rm_war] [/rm] [equinoxbridgeservlet]: Destroy successful.
[7/1/15 17:33:46:018 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [rm_war] [/rm] [addons]: Destroy successful.
[7/1/15 17:33:46:034 SGT] 00000004 ApplicationMg A   WSVR0220I: Application stopped: rm_war
[7/1/15 17:33:46:034 SGT] 00000004 ApplicationMg A   WSVR0217I: Stopping application: qm_war
[7/1/15 17:33:57:671 SGT] 000000c3 CoordinatorCo W   HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 6 seconds.
[7/1/15 17:34:00:292 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [qm_war] [/qm] [equinoxbridgeservlet]: Destroy successful.
[7/1/15 17:34:00:323 SGT] 00000004 ApplicationMg A   WSVR0220I: Application stopped: qm_war
[7/1/15 17:34:00:323 SGT] 00000004 ApplicationMg A   WSVR0217I: Stopping application: ccm_war
[7/1/15 17:34:10:151 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [ccm_war] [/ccm] [equinoxbridgeservlet]: Destroy successful.
[7/1/15 17:34:10:229 SGT] 00000004 ApplicationMg A   WSVR0220I: Application stopped: ccm_war
[7/1/15 17:34:10:245 SGT] 00000004 ApplicationMg A   WSVR0217I: Stopping application: ibmasyncrsp
[7/1/15 17:34:10:245 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [ibmasyncrsp] [/IBM_WS_SYS_RESPONSESERVLET] [rspservlet]: Destroy successful.
[7/1/15 17:34:10:245 SGT] 00000004 ApplicationMg A   WSVR0220I: Application stopped: ibmasyncrsp
[7/1/15 17:34:10:245 SGT] 00000004 ApplicationMg A   WSVR0217I: Stopping application: admin_war
[7/1/15 17:34:11:009 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [admin_war] [/admin] [equinoxbridgeservlet]: Destroy successful.
[7/1/15 17:34:11:103 SGT] 00000004 ApplicationMg A   WSVR0220I: Application stopped: admin_war
[7/1/15 17:34:11:103 SGT] 00000004 ApplicationMg A   WSVR0217I: Stopping application: DefaultApplication
[7/1/15 17:34:11:103 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [DefaultApplication] [/] [Snoop Servlet]: Destroy successful.
[7/1/15 17:34:11:103 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [DefaultApplication] [/] [Hello Pervasive Servlet]: Destroy successful.
[7/1/15 17:34:11:118 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [DefaultApplication] [/] [Hit Count Servlet]: Destroy successful.
[7/1/15 17:34:11:165 SGT] 00000004 EJBContainerI I   WSVR0041I: Stopping EJB jar: Increment.jar
[7/1/15 17:34:11:181 SGT] 00000004 EJBContainerI I   WSVR0059I: EJB jar stopped: Increment.jar
[7/1/15 17:34:11:181 SGT] 00000004 ApplicationMg A   WSVR0220I: Application stopped: DefaultApplication
[7/1/15 17:34:11:181 SGT] 00000004 ApplicationMg A   WSVR0217I: Stopping application: converter_war
[7/1/15 17:34:11:524 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [converter_war] [/converter] [equinoxbridgeservlet]: Destroy successful.
[7/1/15 17:34:11:524 SGT] 00000004 ApplicationMg A   WSVR0220I: Application stopped: converter_war
[7/1/15 17:34:11:524 SGT] 00000004 ApplicationMg A   WSVR0217I: Stopping application: clmhelp_war
[7/1/15 17:34:17:124 SGT] 00000004 servlet       I com.ibm.ws.webcontainer.servlet.ServletWrapper doDestroy SRVE0253I: [clmhelp_war] [/clmhelp] [equinoxbridgeservlet]: Destroy successful.
[7/1/15 17:34:17:187 SGT] 00000004 ApplicationMg A   WSVR0220I: Application stopped: clmhelp_war
[7/1/15 17:34:17:202 SGT] 00000004 SchedulerServ I   SCHD0040I: The Scheduler Service is stopping.
[7/1/15 17:34:17:202 SGT] 00000004 SchedulerServ I   SCHD0002I: The Scheduler Service has stopped.
[7/1/15 17:34:17:218 SGT] 00000004 AppProfileCom I   ACIN0009I: The application profiling service is stopping.
[7/1/15 17:34:17:218 SGT] 00000004 ActivitySessi I   WACS0049I: The ActivitySession service is stopping.
[7/1/15 17:34:17:218 SGT] 00000004 ObjectPoolSer I   OBPL0011I: The Object Pool service is stopping.
[7/1/15 17:34:17:233 SGT] 00000004 distSecurityC I   securityServiceStarted is false
[7/1/15 17:34:17:249 SGT] 00000004 CGBridgeSubsc I   CWRCB0104I: The core group bridge service has stopped the subscription router.
[7/1/15 17:34:17:249 SGT] 00000004 CGBridgeServi I   CWRCB0103I: The core group bridge service has stopped.
[7/1/15 17:34:17:249 SGT] 00000004 DCSStackImpl  I   DCSV8011I: DCS Stack DefaultCoreGroup at Member CBCRTCell01\CBCRTNode01\server1: Member received a request for shutdown.
[7/1/15 17:34:17:311 SGT] 00000004 TCPChannel    I   TCPC0002I: TCP Channel TCPInboundChannel_ipcc.Default_IPC_Connector_Name has stopped listening on host CBCRT.cbcmain.cbc.chinabank.ph  (IPv4: 127.0.0.1) port 9633.
[7/1/15 17:34:17:405 SGT] 00000004 FailureScopeC A   WTRN0105I: The transaction service has shutdown successfully with no transactions requiring recovery.
[7/1/15 17:34:17:421 SGT] 00000004 TCPChannel    I   TCPC0002I: TCP Channel TCP_5 has stopped listening on host *  (IPv6) port 9353.
[7/1/15 17:34:17:421 SGT] 00000004 ServerCollabo A   WSVR0024I: Server server1 stopped

server1_73c273c2_15.07.01_17.31.26.2741905349994276584417.txt content:

[7/1/15 17:31:26:274 SGT]     FFDC Exception:javax.net.ssl.SSLException SourceId:com.ibm.ws.ssl.channel.impl.SSLReadServiceContext ProbeId:192 Reporter:com.ibm.ws.ssl.channel.impl.SSLReadServiceContext@2f2f2f2f
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
    at com.ibm.jsse2.o.a(o.java:17)
    at com.ibm.jsse2.SSLEngineImpl.a(SSLEngineImpl.java:424)
    at com.ibm.jsse2.SSLEngineImpl.a(SSLEngineImpl.java:201)
    at com.ibm.jsse2.SSLEngineImpl.j(SSLEngineImpl.java:276)
    at com.ibm.jsse2.SSLEngineImpl.b(SSLEngineImpl.java:520)
    at com.ibm.jsse2.SSLEngineImpl.a(SSLEngineImpl.java:510)
    at com.ibm.jsse2.SSLEngineImpl.unwrap(SSLEngineImpl.java:527)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:2)
    at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext.decryptMessage(SSLReadServiceContext.java:1218)
    at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext.read(SSLReadServiceContext.java:526)
    at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext.read(SSLReadServiceContext.java:382)
    at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.readyInboundPostHandshake(SSLConnectionLink.java:715)
    at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyHandshakeCompletedCallback.complete(SSLConnectionLink.java:413)
    at com.ibm.ws.ssl.channel.impl.SSLUtils.handleHandshake(SSLUtils.java:1066)
    at com.ibm.ws.ssl.channel.impl.SSLHandshakeIOCallback.complete(SSLHandshakeIOCallback.java:87)
    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
    at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)

==> Performing default dump from com.ibm.ws.ssl.core.SSLDiagnosticModule :Wed Jul 01 17:31:26 SGT 2015
SSLConfigManager configuration:
NodeDefaultSSLSettings===SSLConfig.toString() {
com.ibm.ssl.clientAuthenticationSupported = false
com.ibm.ssl.changed = 9
com.ibm.ssl.contextProvider = IBMJSSE2
com.ibm.ssl.trustStoreProvider = IBMJCE
com.ibm.ssl.protocol = SSL_TLS
com.ibm.ssl.keyStoreReadOnly = false
com.ibm.ssl.alias = NodeDefaultSSLSettings
com.ibm.ssl.keyStoreCreateCMSStash = false
com.ibm.ssl.securityLevel = HIGH
com.ibm.ssl.trustStoreName = CellDefaultTrustStore
com.ibm.ssl.configURLLoadedFrom = security.xml
hostname = localhost
com.ibm.ssl.trustStorePassword = *****
com.ibm.ssl.keyStoreUseForAcceleration = false
com.ibm.ssl.trustManager = IbmPKIX|IBMJSSE2
soReuseAddr = 1
com.ibm.ssl.validationEnabled = false
com.ibm.ssl.trustStoreInitializeAtStartup = false
com.ibm.ssl.keyManager = IbmX509|IBMJSSE2
com.ibm.ssl.keyStoreFileBased = true
endPointName = IPC_CONNECTOR_ADDRESS
com.ibm.ssl.keyStoreType = PKCS12
com.ibm.ssl.trustStoreFileBased = true
com.ibm.ssl.trustStoreCreateCMSStash = false
com.ibm.ssl.trustStoreScope = (cell):CBCRTCell01
com.ibm.ssl.trustStore = D:/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/CBCRTCell01/trust.p12
com.ibm.ssl.keyStoreProvider = IBMJCE
com.ibm.ssl.daysBeforeExpireWarning = 60
com.ibm.ssl.clientAuthentication = false
com.ibm.ssl.keyStore = D:/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/CBCRTCell01/nodes/CBCRTNode01/key.p12
port = 9633
com.ibm.ssl.trustStoreUseForAcceleration = false
com.ibm.ssl.trustStoreReadOnly = false
com.ibm.ssl.keyStoreScope = (cell):CBCRTCell01:(node):CBCRTNode01
com.ibm.ssl.tokenEnabled = false
com.ibm.ssl.keyStoreName = NodeDefaultKeyStore
newConnectionBufferSize = 8192
com.ibm.ssl.keyStorePassword = *****
com.ibm.ssl.keyStoreInitializeAtStartup = false
inactivityTimeout = 300
com.ibm.ssl.trustStoreType = PKCS12
}CellDefaultSSLSettings===SSLConfig.toString() {
com.ibm.ssl.changed = 4
com.ibm.ssl.keyStorePassword = *****
com.ibm.ssl.keyStoreScope = (cell):CBCRTCell01
com.ibm.ssl.daysBeforeExpireWarning = 60
com.ibm.ssl.trustStoreFileBased = true
com.ibm.ssl.keyStoreName = CellDefaultKeyStore
com.ibm.ssl.keyStoreUseForAcceleration = false
com.ibm.ssl.trustStoreReadOnly = false
com.ibm.ssl.contextProvider = IBMJSSE2
com.ibm.ssl.keyStoreFileBased = true
com.ibm.ssl.alias = CellDefaultSSLSettings
com.ibm.ssl.keyManager = IbmX509|IBMJSSE2
com.ibm.ssl.keyStore = D:/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/CBCRTCell01/key.p12
com.ibm.ssl.trustStoreInitializeAtStartup = false
com.ibm.ssl.keyStoreType = PKCS12
com.ibm.ssl.clientAuthentication = false
com.ibm.ssl.trustStoreProvider = IBMJCE
com.ibm.ssl.keyStoreInitializeAtStartup = false
com.ibm.ssl.trustStore = D:/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/CBCRTCell01/trust.p12
com.ibm.ssl.tokenEnabled = false
com.ibm.ssl.trustManager = IbmPKIX|IBMJSSE2
com.ibm.ssl.protocol = SSL_TLS
com.ibm.ssl.trustStorePassword = *****
com.ibm.ssl.trustStoreName = CellDefaultTrustStore
com.ibm.ssl.keyStoreCreateCMSStash = false
com.ibm.ssl.trustStoreCreateCMSStash = false
com.ibm.ssl.keyStoreReadOnly = false
com.ibm.ssl.trustStoreUseForAcceleration = false
com.ibm.ssl.securityLevel = HIGH
com.ibm.ssl.trustStoreType = PKCS12
com.ibm.ssl.trustStoreScope = (cell):CBCRTCell01
com.ibm.ssl.validationEnabled = false
com.ibm.ssl.clientAuthenticationSupported = false
com.ibm.ssl.keyStoreProvider = IBMJCE
com.ibm.ssl.configURLLoadedFrom = security.xml
}*** INBOUND ***

Cell Alias: CellDefaultSSLSettings
NodeGroup Alias(es): {}
Node Alias: NodeDefaultSSLSettings
Server Alias: null
Cluster Alias: null
EndPoint Alias(es): {}
Effective Process Alias: NodeDefaultSSLSettings
Effective Process Scope: node

*** OUTBOUND ***

Cell Alias: CellDefaultSSLSettings
NodeGroup Alias(es): {}
Node Alias: NodeDefaultSSLSettings
Server Alias: null
Cluster Alias: null
EndPoint Alias(es): {}
Effective Process Alias: NodeDefaultSSLSettings
Effective Process Scope: node:javax.net.ssl.SSLException: Received fatal alert: unknown_ca
+Data for directive [defaultsslcomponentimpl] obtained.:
==> Dump complete for com.ibm.ws.ssl.core.SSLDiagnosticModule :Wed Jul 01 17:31:26 SGT 2015

3 answers



permanent link
Tiffany Pei (139117) | answered Jul 01 '15, 9:51 p.m.
What's the contents of D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\ffdc\server1_73c273c2_15.07.01_17.31.26.2741905349994276584417.txt?

Comments
Irell Carloe Santos commented Jul 01 '15, 9:56 p.m.

please see in the edited question. Thanks.


permanent link
Howard Hsiao (5.4k7) | answered Jul 01 '15, 11:19 p.m.
JAZZ DEVELOPER

You have got SSLException with unknown_ca.
One possible cause is that there are problems with the client certificate used by you.
Does your certificate have any extensions?
If it does, please create a self signed certificate without any extensions and used it as your client certificate.

Comments
Irell Carloe Santos commented Jul 02 '15, 9:07 p.m.

Hi Howard,

Thanks. Will try to look into this error.


permanent link
Donald Nong (14.4k213) | answered Jul 02 '15, 2:58 a.m.
The second line of the log content that you post is this
[7/1/15 17:32:54:741 SGT] 000000a6 AdminHelper   A   ADMN1020I: An attempt is made to stop the server1 server. (User ID = defaultWIMFileBasedRealm/wasadm)
That means the WAS server was stopped by the use "wasadm". I am not sure whether this is part of the procedure of restarting the server. If so, the rest of the log is not worth reading. In effect, you only provide one line (the first line) of message for others to figure out the cause.

You need to explain clearly and in details what "stopped working" means. The best way to investigate the issue is to contact Support so that all relevant log files can be examined.
https://jazz.net/help/support/

Comments
Irell Carloe Santos commented Jul 02 '15, 5:16 a.m.

That was actually the problem, we did not try to stop the service. We are using RTC then suddenly  the webpage becomes unavailable, I'm thinking It was caused by some other thing that's why I posted the logs. The first line of the message is
[7/1/15 17:31:26:274 SGT] 00000042 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\ffdc\server1_73c273c2_15.07.01_17.31.26.2741905349994276584417.txt com.ibm.ws.ssl.channel.impl.SSLReadServiceContext 192 - Do you know something about this line? Thanks.


Howard Hsiao commented Jul 02 '15, 7:22 a.m. | edited Jul 02 '15, 7:25 a.m.
JAZZ DEVELOPER

Please check my previous comments about that error.


Donald Nong commented Jul 02 '15, 9:43 p.m.

The SSL error indicates a "client certificate" error "unknown_ca", as Howard pointed out. However, since the server complains about "unknown_ca", it means that it refuses to accept the client certificate. This can appear when you use a self-signed certificate as your client certificate. To fix this, you need to add the client certificate's CA to the trust store of the server, so that the server recognize the CA when receiving the client certificate. Not knowing your configuration, I cannot give detailed steps, but you can search with keyword "WebSphere trust store SSO" and get familiar with the topic.

Before you try to anything about it, you may want to determine which client sends out the not-so-good certificate. If it's someone trying to hack into your system, you definitely don't want to trust that certificate's CA at all.

Your answer


Register or to post your answer.