About 1 month after we upgraded our whole environment to 4.0.5 (from 4.0.1), we have been seeing almost 100% CPU usage on our RRC server. The CPU is being used by the RRC java process. Immediately after a restart of the application, CPU is low, but a couple hours afterwards the CPU goes and stays high. Checking the rm.log, we can only find one message around the time of the issue starting (see below).  (we have seen the same message shortly after our recent restarts)

It would not appear to be something a user is doing, as the CPU stays consistently high even when there are no Licenses being used by the application. Memory usage on the system seems to be just over 50%.

Any idea when may be causing this? How we can figure it out and fix it?


2014-04-05 01:22:46,719 [  RM-ClusterSafeInitializers-3] ERROR .recentfeeds.internal.EntryFactory$PopulateInitial  - Exception occurred while getting the recent resources details
com.ibm.rdm.fronting.server.exception.InternalServerErrorException: Propagated exception; original message [Read timed out]
        at com.ibm.rdm.fronting.server.exception.RRSException.newException(RRSException.java:220)
        at com.ibm.rdm.fronting.server.exception.RRSException.newException(RRSException.java:737)
        at com.ibm.rdm.fronting.server.exception.RRSException.capture(RRSException.java:741)
        at com.ibm.rdm.fronting.server.core.jpi.internal.InternalServiceExchange.checkError(InternalServiceExchange.java:53)
        at com.ibm.rdm.fronting.server.core.jpi.internal.InternalServiceExchange.getResponse(InternalServiceExchange.java:37)
        at com.ibm.rdm.fronting.server.core.jpi.internal.CoreServiceRequest.executeServiceRequest(CoreServiceRequest.java:63)
        at com.ibm.rdm.fronting.server.core.spi.BaseServiceRequest.execute(BaseServiceRequest.java:117)
        at com.ibm.rdm.fronting.server.services.query.internal.QueryServiceUtil.doQuerySparql(QueryServiceUtil.java:2009)
        at com.ibm.rdm.fronting.server.services.query.internal.QueryServiceUtil.querySparql(QueryServiceUtil.java:1830)
        at com.ibm.rdm.fronting.server.services.recentfeeds.internal.EntryFactory$PopulateInitial.run(EntryFactory.java:108)
        at com.ibm.rdm.fronting.server.services.recentfeeds.internal.ClusterSafeInitializer$InternalInit.call(ClusterSafeInitializer.java:48)
        at com.ibm.rdm.fronting.server.concurrent.RRSConcurrency$ForwardingListeningExecutorServiceExtension$1.call(RRSConcurrency.java:106)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
        at java.util.concurrent.FutureTask.run(FutureTask.java:149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931)
        at java.lang.Thread.run(Thread.java:773)
Caused by: com.ibm.rdm.fronting.server.exception.RRSException: Read timed out
        at com.ibm.rdm.fronting.server.core.jpi.internal.AbstractCoreService.sendHttpRequest(AbstractCoreService.java:115)
        at com.ibm.rdm.fronting.server.core.jpi.internal.AbstractCoreService.sendRequest(AbstractCoreService.java:81)
        at com.ibm.rdm.fronting.server.core.jpi.internal.InternalCoreServices.invokeHttpRequest(InternalCoreServices.java:860)
        at com.ibm.rdm.fronting.server.core.jpi.internal.InternalCoreServices.handleCoreBulkables(InternalCoreServices.java:790)
        at com.ibm.rdm.fronting.server.core.jpi.internal.InternalCoreServices.handleCoreRequests(InternalCoreServices.java:726)
        at com.ibm.rdm.fronting.server.core.jpi.internal.InternalCoreServices.execute(InternalCoreServices.java:310)
        at com.ibm.rdm.fronting.server.core.jpi.internal.InternalCoreServices.execute(InternalCoreServices.java:242)
        at com.ibm.rdm.fronting.server.core.jpi.internal.InternalCoreServices.execute(InternalCoreServices.java:234)
        at com.ibm.rdm.fronting.server.core.jpi.internal.CoreServiceRequest.executeServiceRequest(CoreServiceRequest.java:61)
        ... 11 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:140)
        at com.ibm.jsse2.b.a(b.java:197)
        at com.ibm.jsse2.b.a(b.java:242)
        at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:750)
        at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:484)
        at com.ibm.jsse2.f.read(f.java:49)
        at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:149)
        at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:110)
        at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:264)
        at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
        at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281)
        at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
        at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)
        at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at com.ibm.rdm.fronting.server.core.jpi.internal.http.StatsConnectionManager$RequestLineInterceptor.invoke(StatsConnectionManager.java:286)
        at com.ibm.rdm.fronting.server.core.jpi.internal.http.$Proxy29.receiveResponseHeader(Unknown Source)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:464)
        at com.ibm.rdm.fronting.server.core.jpi.internal.http.RRCJazzHttpClient$1.execute(RRCJazzHttpClient.java:223)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
        at com.ibm.team.jfs.app.http.client.JazzHttpClient.execute(JazzHttpClient.java:288)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:927)
        at com.ibm.rdm.fronting.server.core.jpi.internal.AbstractCoreService.sendHttpRequest(AbstractCoreService.java:105)
        ... 19 more
Request
        --Not Provided--
Response
        --Not Provided--