Why do I get "No such file or directory" errors when loading a repository workspace?

Hi,

I have a customer using RTC 6.0.2.

They recently started getting load errors with their repository workspaces, which then results in a out-of-sync sandbox.

After investigating, I found 5 files that are failing to load. Browsing these files via the web client results in Internal Server Error 500.

In the ccm.log I see the following error:

2018-01-23 12:00:25,799 [WebContainer : 29 @@ 12:00 clmadmin <Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0@10.10.1.11> /ccm/service/com.ibm.team.filesystem.service.internal.rest.IFilesystemContentService/_y8ElkHpkEea0B8k3wlr4Gw/_9z_UMB5fEeWzYfrVtIYpgw/Common.jar] ERROR com.ibm.team.filesystem.rest                        - CRRTC5043I: Error occurred

com.ibm.team.repository.common.InternalRepositoryException: No such file or directory

    at com.ibm.team.scm.service.internal.content.VersionedContentService$DeltaUncompressContent.handleContent(VersionedContentService.java:4776)

    at com.ibm.team.scm.service.internal.content.VersionedContentService$ContentToRandomAccessContentSource.receiveContent(VersionedContentService.java:5077)

    at com.ibm.team.scm.service.internal.content.VersionedContentService.fetchContent(VersionedContentService.java:3590)

    at com.ibm.team.scm.service.internal.content.VersionedContentService.uncompressToStream(VersionedContentService.java:3493)

    at com.ibm.team.scm.service.internal.content.VersionedContentService.access$1(VersionedContentService.java:3476)

    at com.ibm.team.scm.service.internal.content.VersionedContentService$ContentDirectToStream.receiveContent(VersionedContentService.java:4942)

    at com.ibm.team.scm.service.internal.content.VersionedContentService.fetchContent(VersionedContentService.java:3590)

    at com.ibm.team.scm.service.internal.content.VersionedContentService.doRetrieveContent(VersionedContentService.java:3392)

    at com.ibm.team.scm.service.internal.content.VersionedContentService.access$27(VersionedContentService.java:3387)

    at com.ibm.team.scm.service.internal.content.VersionedContentService$13.run(VersionedContentService.java:2442)

    at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase$Transaction.run(RepositoryDatabase.java:570)

    at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase$1.run(RepositoryDatabase.java:345)

    at com.ibm.team.repository.service.internal.rdb.ConnectionPoolService.withNewDeferredConnection(ConnectionPoolService.java:551)

    at com.ibm.team.repository.service.internal.rdb.ConnectionPoolService.withCurrentDeferredConnection(ConnectionPoolService.java:569)

    at sun.reflect.GeneratedMethodAccessor324.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

    at java.lang.reflect.Method.invoke(Method.java:611)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)

    at com.sun.proxy.$Proxy917.withCurrentDeferredConnection(Unknown Source)

    at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase.runTransaction(RepositoryDatabase.java:342)

    at com.ibm.team.repository.service.internal.rdb.RepositoryDatabase.runInTransaction(RepositoryDatabase.java:291)

    at com.ibm.team.repository.service.internal.PrimitiveTransactionService.runInTransaction(PrimitiveTransactionService.java:103)

    at sun.reflect.GeneratedMethodAccessor319.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

    at java.lang.reflect.Method.invoke(Method.java:611)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)

    at com.sun.proxy.$Proxy1190.runInTransaction(Unknown Source)

    at com.ibm.team.repository.service.internal.TransactionService.runInTransaction(TransactionService.java:45)

    at sun.reflect.GeneratedMethodAccessor530.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

    at java.lang.reflect.Method.invoke(Method.java:611)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)

    at com.sun.proxy.$Proxy2063.runInTransaction(Unknown Source)

    at com.ibm.team.scm.service.internal.content.VersionedContentService.runInPossiblyExistingTransaction(VersionedContentService.java:2419)

    at com.ibm.team.scm.service.internal.content.VersionedContentService.fetchContentTrusted(VersionedContentService.java:2439)

    at sun.reflect.GeneratedMethodAccessor3435.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

    at java.lang.reflect.Method.invoke(Method.java:611)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)

    at com.sun.proxy.$Proxy3146.fetchContentTrusted(Unknown Source)

    at com.ibm.team.scm.service.internal.content.ComponentSpecificVersionedContentService.fetchContentTrusted(ComponentSpecificVersionedContentService.java:67)

    at com.ibm.team.filesystem.service.internal.rest.FilesystemContentService.perform_GET(FilesystemContentService.java:272)

    at com.ibm.team.repository.service.TeamRawService.service(TeamRawService.java:84)

    at sun.reflect.GeneratedMethodAccessor3487.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

    at java.lang.reflect.Method.invoke(Method.java:611)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.invoke(ExportProxyServiceRecord.java:361)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord.access$0(ExportProxyServiceRecord.java:347)

    at org.eclipse.soda.sat.core.internal.record.ExportProxyServiceRecord$ExportedServiceInvocationHandler.invoke(ExportProxyServiceRecord.java:56)

    at com.sun.proxy.$Proxy3248.service(Unknown Source)

    at com.ibm.team.repository.servlet.AbstractTeamServerServlet.doRestService(AbstractTeamServerServlet.java:971)

    at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest2(AbstractTeamServerServlet.java:2520)

    at com.ibm.team.repository.servlet.AbstractTeamServerServlet.handleRequest(AbstractTeamServerServlet.java:2315)

    at com.ibm.team.repository.servlet.AbstractTeamServerServlet.service(AbstractTeamServerServlet.java:1794)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)

    at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(ServletManager.java:180)

    at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)

    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:126)

    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:76)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)

    at org.eclipse.equinox.servletbridge.BridgeServlet.service(BridgeServlet.java:138)

    at com.ibm.team.repository.server.servletbridge.JazzServlet.service(JazzServlet.java:74)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)

    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1227)

    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:776)

    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:458)

    at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)

    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)

    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)

    at com.ibm.team.repository.server.servletbridge.BridgeFilter.processDelegate(BridgeFilter.java:165)

    at com.ibm.team.repository.server.servletbridge.BridgeFilter.doFilter(BridgeFilter.java:198)

    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)

    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)

    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)

    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)

    at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3761)

    at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)

    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:976)

    at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)

    at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)

    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)

    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)

    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)

    at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)

    at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)

    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)

    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)

    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)

    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)

    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)

    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)

    at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)

    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)

Caused by: java.io.IOException: No such file or directory

    at java.io.File.checkAndCreate(File.java:1716)

    at java.io.File.createTempFile0(File.java:1737)

    at java.io.File.createTempFile(File.java:1814)

    at com.ibm.team.scm.service.internal.content.InputStreamContentSource.createTempFile(InputStreamContentSource.java:278)

    at com.ibm.team.scm.service.internal.content.InputStreamContentSource.createAndOpenTempFile(InputStreamContentSource.java:283)

    at com.ibm.team.scm.service.internal.content.InputStreamContentSource.readStreamUpTo(InputStreamContentSource.java:312)

    at com.ibm.team.scm.service.internal.content.InputStreamContentSource.readBuffer(InputStreamContentSource.java:103)

    at com.ibm.team.scm.service.internal.content.InputStreamContentSource.internalOffsetCopyBytesFromDisk(InputStreamContentSource.java:181)

    at com.ibm.team.scm.service.internal.content.InputStreamContentSource.offsetCopyBytes(InputStreamContentSource.java:214)

    at com.ibm.team.scm.service.internal.content.ContentStorageDelta.apply(ContentStorageDelta.java:125)

    at com.ibm.team.scm.service.internal.content.VersionedContentService$DeltaUncompressContent.handleContent(VersionedContentService.java:4774)

    ... 100 more

3 of these files where fixed by deleting them directly from the stream (using the web client) and then re-adding them into the workspace, then delivering back to the stream.

However, the remaining 2 files (both are .jar) still throw the error, even after deleting and re-adding several times.

If I leave these 2 files deleted, then the workspace loads successfully.

How can I resolve this issue?

Thank you,
Sudheer