The specified network name is no longer available

I'm getting the following in the Eclipse client when I search for workitems using text. If I search by id, it works:

com.ibm.team.repository.common.TeamRepositoryException: CRJAZ8201E: Could not execute query ( )
at com.ibm.team.fulltext.service.internal.query.QueryManagerImpl.find(QueryManagerImpl.java:257)
at com.ibm.team.fulltext.service.internal.FullTextServiceImpl.find(FullTextServiceImpl.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at $Proxy91.find(null)
at com.ibm.team.workitem.service.internal.query.ASTTransformer.createFullTextPredicate(ASTTransformer.java:397)
at com.ibm.team.workitem.service.internal.query.ASTTransformer.createExpressionPredicate(ASTTransformer.java:372)
at com.ibm.team.workitem.service.internal.query.ASTTransformer.visit(ASTTransformer.java:298)
at com.ibm.team.workitem.common.expression.AttributeExpression.accept(AttributeExpression.java:126)
at com.ibm.team.workitem.service.internal.query.ASTTransformer.visit(ASTTransformer.java:247)
at com.ibm.team.workitem.common.expression.Term.accept(Term.java:143)
at com.ibm.team.workitem.common.expression.Statement.accept(Statement.java:91)
at com.ibm.team.workitem.service.internal.query.ASTTransformer.transform(ASTTransformer.java:186)
at com.ibm.team.workitem.service.internal.query.QueryRepositoryService.fetchQueryResults(QueryRepositoryService.java:181)
at sun.reflect.GeneratedMethodAccessor212.invoke(null)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:370)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:356)
at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)
at $Proxy187.fetchQueryResults(null)
at sun.reflect.GeneratedMethodAccessor212.invoke(null)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleMethod(AbstractTeamServerServlet.java:1170)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.executeMethod(AbstractTeamServerServlet.java:926)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.doPost(AbstractTeamServerServlet.java:728)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:1773)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:1642)
at com.ibm.team.repository.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1555)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:180)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:75)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at org.eclipse.equinox.servletbridge.BridgeServlet.service(BridgeServlet.java:121)
at com.ibm.team.repository.server.servletbridge.JazzServlet.service(JazzServlet.java:54)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:445)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:504)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:301)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1772)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:816)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1563)
at com.ibm.team.repository.common.internal.marshal.util.MarshallerUtil.decodeExceptions(MarshallerUtil.java:326)
at com.ibm.team.repository.common.internal.marshal.util.MarshallerUtil.decodeExceptions(MarshallerUtil.java:296)
at com.ibm.team.repository.common.internal.marshal.util.MarshallerUtil.decodeFault(MarshallerUtil.java:261)
at com.ibm.team.repository.transport.client.RemoteTeamService.constructExceptionFromFault(RemoteTeamService.java:613)
at com.ibm.team.repository.transport.client.RemoteTeamService.executeMethod(RemoteTeamService.java:483)
at com.ibm.team.repository.transport.client.RemoteTeamService.invoke(RemoteTeamService.java:201)
at com.ibm.team.repository.transport.client.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:43)
at $Proxy91.fetchQueryResults(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at com.ibm.team.repository.client.internal.ServiceInterfaceProxy.invokeServiceCall(ServiceInterfaceProxy.java:149)
at com.ibm.team.repository.client.internal.ServiceInterfaceProxy.invoke(ServiceInterfaceProxy.java:84)
at $Proxy91.fetchQueryResults(Unknown Source)
at com.ibm.team.workitem.common.internal.query.QueryCommon$3.run(QueryCommon.java:155)
at com.ibm.team.workitem.common.internal.query.QueryCommon$3.run(QueryCommon.java:1)
at com.ibm.team.workitem.client.internal.ClientServiceContext$1.run(ClientServiceContext.java:41)
at com.ibm.team.repository.client.internal.TeamRepository$3.run(TeamRepository.java:1169)
at com.ibm.team.repository.common.transport.CancelableCaller.call(CancelableCaller.java:79)
at com.ibm.team.repository.client.internal.TeamRepository.callCancelableService(TeamRepository.java:1162)
at com.ibm.team.workitem.client.internal.ClientServiceContext.callCancelableService(ClientServiceContext.java:45)
at com.ibm.team.workitem.common.internal.query.QueryCommon.fetchQueryResults(QueryCommon.java:152)
at com.ibm.team.workitem.common.internal.query.QueryResultIterator.fetchFirstPage(QueryResultIterator.java:151)
at com.ibm.team.workitem.common.internal.query.QueryResultIterator.update(QueryResultIterator.java:129)
at com.ibm.team.workitem.common.internal.query.QueryResultIterator.getResultSize(QueryResultIterator.java:113)
at com.ibm.team.workitem.rcp.ui.internal.viewer.QueryWorkItemRetrievalStrategy.retrieveRootsBackground(QueryWorkItemRetrievalStrategy.java:93)
at com.ibm.team.workitem.rcp.ui.internal.viewer.QueryWorkItemRetrievalStrategy$1.retrieve(QueryWorkItemRetrievalStrategy.java:72)
at com.ibm.team.workitem.rcp.ui.internal.viewer.ResultRetrievalStrategy$RetrievalTask.run(ResultRetrievalStrategy.java:66)
at com.ibm.team.workitem.rcp.ui.internal.viewer.ResultRetrievalStrategy$RetrievalJob.runProtected(ResultRetrievalStrategy.java:112)
at com.ibm.team.foundation.client.util.FoundationJob.run(FoundationJob.java:68)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Caused by: java.io.IOException: The specified network name is no longer available.
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:333)
at org.apache.lucene.store.FSIndexInput.readInternal(FSDirectory.java:536)
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:74)
at org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:220)
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:93)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:34)
at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:57)
at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:62)
at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:118)
at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:149)
at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:154)
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:148)
at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:358)
at org.apache.lucene.index.MultiReader.docFreq(MultiReader.java:209)
at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:79)
at org.apache.lucene.search.MultiSearcher.docFreq(MultiSearcher.java:140)
at org.apache.lucene.search.Similarity.idf(Similarity.java:457)
at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:44)
at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:145)
at org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:99)
at org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:161)
at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:187)
at org.apache.lucene.search.BooleanQuery$BooleanWeight2.<init>(BooleanQuery.java:342)
at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384)
at org.apache.lucene.search.Query.weight(Query.java:95)
at org.apache.lucene.search.Hits.<init>(Hits.java:42)
at org.apache.lucene.search.Searcher.search(Searcher.java:45)
at com.ibm.team.fulltext.service.internal.index.IndexManagerImpl$1.search(IndexManagerImpl.java:509)
at com.ibm.team.fulltext.service.internal.query.QueryManagerImpl.execute(QueryManagerImpl.java:387)
at com.ibm.team.fulltext.service.internal.query.QueryManagerImpl.find(QueryManagerImpl.java:255)
... 97 more

and the following on the server:

com.ibm.ws.webcontainer.srt.SRTServletRequest parseParameters SRVE0133E: An error occurred while parsing parameters. {0}
java.io.IOException: Async IO operation failed (3), reason: RC: 55 The specified network resource or device is no longer available.

at com.ibm.io.async.AsyncLibrary$IOExceptionCache.<init>(AsyncLibrary.java:891)
at com.ibm.io.async.AsyncLibrary$IOExceptionCache.get(AsyncLibrary.java:904)
at com.ibm.io.async.AsyncLibrary.getIOException(AsyncLibrary.java:918)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:705)
at com.ibm.io.async.ResultHandler$CompletionProcessingRunnable.run(ResultHandler.java:934)
at java.lang.Thread.run(Thread.java:736)

Not sure if they are related, but once this problem occurs only a server restart will correct. Any ideas?

Hi,

Could sound like the location of your work item index is wrong or not available. Check your team server.properties for this setting and validate that you can access the indeces.

Hi,

you could also recreate the index file using repotools -reindex and -rebuildTextIndices.

The confusing issue is though that the symptoms go away when restarting the server. Are there any additional indicators in the logs?

Ralph

Hi,

you could also recreate the index file using repotools -reindex and -rebuildTextIndices.

The confusing issue is though that the symptoms go away when restarting the server. Are there any additional indicators in the logs?

Ralph

I didn't see any other errors that could apply; but I think we found the problem: the network connection to the SAN where the index files are was getting interrupted, we changed the route of network connection to make it more reliable. It seems to be resolved.

Because we have this setup in a primary/failover configuration, the index files are stored on the SAN where either server can access them. I understand the network connection should be reliable, but if there is a bandwidth problem (i.e. unusual delay) or some other network issue, and the connection to the SAN is temporally interrupted, you would think the connection could be reestablished to the index files reliably?

It appears, if any interruption in the connection from the Jazz server to index files, this error would occur and the Jazz server could not reestablish the connection without a restart.

Bi Bob,

you might want to file an enhancement request for a more gracefull recovery.

Thanks,

Ralph

I didn't see any other errors that could apply; but I think we found the problem: the network connection to the SAN where the index files are was getting interrupted, we changed the route of network connection to make it more reliable. It seems to be resolved.

Because we have this setup in a primary/failover configuration, the index files are stored on the SAN where either server can access them. I understand the network connection should be reliable, but if there is a bandwidth problem (i.e. unusual delay) or some other network issue, and the connection to the SAN is temporally interrupted, you would think the connection could be reestablished to the index files reliably?

It appears, if any interruption in the connection from the Jazz server to index files, this error would occur and the Jazz server could not reestablish the connection without a restart.