It's all about the answers!

Ask a question

Is there any fix for this : CRJAZ1165I Marshalling exception processing request for service


Naveen Krishnan (1133) | asked Mar 23 '15, 5:46 p.m.
We are noticing that our RTC server is going down very often. The reason was, the DB user is getting locked, due to multiple number of unsuccessful logins. What I presume is, until this exception throws all the RTC users are trying to revoke the session, which eventually gets failed after this exception is thrown. This is what I understood, please correct me if I am wrong here.

I searched for this, I could not find any fix for this. Please let me know how to fix this issue. Thanks!

CRJAZ1165I Marshalling exception processing request for service "com.ibm.team.repository.common.service.IQueryService".
CRJAZ1170I The request was made by user "xxxx@yy.zz.com" from "xxx.xxx.xxx.xxx".CRJAZ1166I The stack trace hash is 3601A6B65EE2D4314B71A07F3F1E7F90B91A0D97.
com.ibm.team.repository.common.internal.marshal.MarshallingException: Async operation timed out
        at com.ibm.team.repository.common.internal.marshal.impl.EObjectMarshaller.demarshalInputStreamToObject(EObjectMarshaller.java:579)
        at com.ibm.team.repository.common.internal.marshal.impl.EObjectMarshaller.demarshalInputStreamToObject(EObjectMarshaller.java:599)
        at com.ibm.team.repository.common.internal.marshal.impl.WebServicesMarshaller.demarshalInputStreamToServiceRequest(WebServicesMarshaller.java:139)
        at com.ibm.team.repository.servlet.AbstractTeamServerServlet.doPost(AbstractTeamServerServlet.java:690)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
        at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2358)
        at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:2155)
        at com.ibm.team.repository.servlet.AbstractTeamServerServlet.access$0(AbstractTeamServerServlet.java:2140)
        at com.ibm.team.repository.servlet.AbstractTeamServerServlet$1.service(AbstractTeamServerServlet.java:224)
        at com.ibm.team.repository.internal.service.auth.impl.JAuthHandler$1.run(JAuthHandler.java:109)
        at com.ibm.team.repository.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1813)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
        at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:180)
        at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
        at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:126)
        at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:76)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
        at org.eclipse.equinox.servletbridge.BridgeServlet.service(BridgeServlet.java:120)
        at com.ibm.team.repository.server.servletbridge.JazzServlet.service(JazzServlet.java:68)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
   at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
        at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
        at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
        at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
        at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
        at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
        at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)
Caused by: java.net.SocketTimeoutException: Async operation timed out
        at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:189)
        at com.ibm.ws.tcp.channel.impl.TCPReadRequestContextImpl.read(TCPReadRequestContextImpl.java:111)
        at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext.read(SSLReadServiceContext.java:265)
        at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.fillABuffer(HttpServiceContextImpl.java:4171)
        at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readSingleBlock(HttpServiceContextImpl.java:3403)
        at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readBodyBuffer(HttpServiceContextImpl.java:3509)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl.getRequestBodyBuffer(HttpInboundServiceContextImpl.java:1782)
        at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.bufferIsGood(WCCByteBufferInputStream.java:373)
        at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.read(WCCByteBufferInputStream.java:266)
        at com.ibm.ws.webcontainer.srt.http.HttpInputStream.read(HttpInputStream.java:325)
        at java.io.FilterInputStream.read(FilterInputStream.java:127)
        at com.ibm.team.repository.servlet.internal.CountingInputStream.read(CountingInputStream.java:80)
        at com.ibm.team.repository.common.internal.marshal.impl.EObjectMarshaller.demarshalInputStreamToObject(EObjectMarshaller.java:567)
        ... 48 more
Caused by: com.ibm.io.async.AsyncTimeoutException(Async operation timed out, [Timeout, rc=0])
        at com.ibm.io.async.AbstractAsyncFuture.waitForCompletion(AbstractAsyncFuture.java:359)
        at com.ibm.io.async.AsyncFuture.getByteCount(AsyncFuture.java:218)
        at com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.readAIOSync(AioSocketIOChannel.java:215)
        at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:182)


Comments
1
Donald Nong commented Mar 23 '15, 7:10 p.m.

The user "xxxx@yy.zz.com" looks more like an LDAP user, or a JTS/RTC user, not a "DB" user. Are you really sure the issue is caused by the DB user being locked? The DB user is specified in the teamserver.properties file. Unless the DB user is also used somewhere else (other than the RTC server), with a wrong password, I can't imagine how it can get locked during the operation.


Naveen Krishnan commented Mar 25 '15, 9:09 a.m.

Hi Donald, Yes the DB user gets locked, whenever RTC server goes down. As and when RTC goes down, we first see whether the DB user is locked or not, and yes it gets locked, then we unlock that user, from where RTC comes up. When we checked the log, it mentioned that teamserver.properties is having issues, but we didnt see any issue or change there.

And also xxxx@yy.zz.com is the RTC admin user which is configured in LDAP. It is just an admin user which is used for rtc build related purposes.


Donald Nong commented Mar 26 '15, 12:58 a.m.

"It mentioned that teamserver.properties is having issues" - this is where you should go a bit deeper. If the RTC server fails to read the teamserver.properties file, it may get a wrong/corrupted DB user password. If it does work like that, the DB user will for sure get locked when the RTC server repeatedly tries to connect with the wrong/corrupted password.

One answer



permanent link
Naveen Krishnan (1133) | answered Mar 25 '15, 9:09 a.m.
Hi Donald, Yes the DB user gets locked, whenever RTC server goes down. As and when RTC goes down, we first see whether the DB user is locked or not, and yes it gets locked, then we unlock that user, from where RTC comes up. When we checked the log, it mentioned that teamserver.properties is having issues, but we didnt see any issue or change there.

And also xxxx@yy.zz.com is the RTC admin user which is configured in LDAP. It is just an admin user which is used for rtc build related purposes.

Your answer


Register or to post your answer.