We leverage the ant build toolkit to provide RTC users the access to our
in-house build infrastructure. A couple weeks ago a certain build began
encountering this:
[teamFetch] Not using a proxy to reach https://rtp-rtc10:9443/jazz
[teamFetch] Fetching contents of workspace "tbsm_dev Production Build Workspace" to "/bigFS/rtp/FS3/tbsm_dev/prod/TOP".
[teamFetch] com.ibm.team.repository.common.transport.TeamServiceException: CRJAZ0099I HTTP error "java.net.SocketTimeoutException" accessing URL "https://rtp-rtc10:9443/jazz/service/com.ibm.team.filesystem.common.IFilesystemService".
[teamFetch] at com.ibm.team.repository.transport.client.ClientHttpUtil.executeHttpMethod(ClientHttpUtil.java:179)
[teamFetch] at com.ibm.team.repository.transport.client.RemoteTeamService.executeCancelableHttpMethod(RemoteTeamService.java:554)
[teamFetch] at com.ibm.team.repository.transport.client.RemoteTeamService.invokePost(RemoteTeamService.java:527)
[teamFetch] at com.ibm.team.repository.transport.client.RemoteTeamService.executeMethod(RemoteTeamService.java:467)
[teamFetch] at com.ibm.team.repository.transport.client.RemoteTeamService.invoke(RemoteTeamService.java:197)
[teamFetch] at com.ibm.team.repository.transport.client.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:43)
[teamFetch] at $Proxy23.getFileTreeByVersionable(Unknown Source)
[teamFetch] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[teamFetch] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
[teamFetch] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[teamFetch] at java.lang.reflect.Method.invoke(Method.java:618)
[teamFetch] at com.ibm.team.repository.client.internal.ServiceInterfaceProxy.invokeServiceCall(ServiceInterfaceProxy.java:149)
[teamFetch] at com.ibm.team.repository.client.internal.ServiceInterfaceProxy.invoke(ServiceInterfaceProxy.java:84)
[teamFetch] at $Proxy23.getFileTreeByVersionable(Unknown Source)
[teamFetch] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[teamFetch] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
[teamFetch] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[teamFetch] at java.lang.reflect.Method.invoke(Method.java:618)
[teamFetch] at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy$2.run(ScmServiceInterfaceProxy.java:164)
[teamFetch] at com.ibm.team.repository.client.internal.TeamRepository$3.run(TeamRepository.java:1164)
[teamFetch] at com.ibm.team.repository.common.transport.CancelableCaller.call(CancelableCaller.java:79)
[teamFetch] at com.ibm.team.repository.client.internal.TeamRepository.callCancelableService(TeamRepository.java:1157)
[teamFetch] at com.ibm.team.scm.client.internal.ScmClientLibraryContext.callCancelableService(ScmClientLibraryContext.java:70)
[teamFetch] at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy.invokeCancelableService(ScmServiceInterfaceProxy.java:161)
[teamFetch] at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy.invoke(ScmServiceInterfaceProxy.java:91)
[teamFetch] at $Proxy23.getFileTreeByVersionable(Unknown Source)
[teamFetch] at com.ibm.team.filesystem.client.internal.FileSystemServiceProxy.getFileTreeByVersionable(FileSystemServiceProxy.java:262)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.LoadOperation.load(LoadOperation.java:1053)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.LoadOperation$2.run(LoadOperation.java:689)
[teamFetch] at com.ibm.team.filesystem.client.internal.SharingManager.runWithinFileSystemLock(SharingManager.java:730)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.FileSystemOperation.runWithinFileSystemLock(FileSystemOperation.java:173)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.FileSystemOperation.runWithinFileSystemLock(FileSystemOperation.java:153)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.LoadOperation.execute(LoadOperation.java:721)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.FileSystemOperation.run(FileSystemOperation.java:90)
[teamFetch] at com.ibm.team.build.internal.scm.SourceControlUtility.updateFileCopyArea(SourceControlUtility.java:364)
[teamFetch] at com.ibm.team.build.internal.scm.SourceControlUtility.updateFileCopyArea(SourceControlUtility.java:175)
[teamFetch] at com.ibm.team.build.ant.task.TeamFetchTask.doExecute(TeamFetchTask.java:124)
[teamFetch] at com.ibm.team.build.ant.task.AbstractTeamBuildTask.execute(AbstractTeamBuildTask.java:433)
[teamFetch] at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
[teamFetch] at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
[teamFetch] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[teamFetch] at java.lang.reflect.Method.invoke(Method.java:618)
[teamFetch] at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:105)
[teamFetch] at org.apache.tools.ant.Task.perform(Task.java:348)
[teamFetch] at org.apache.tools.ant.Target.execute(Target.java:357)
[teamFetch] at org.apache.tools.ant.Target.performTasks(Target.java:385)
[teamFetch] at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1329)
[teamFetch] at org.apache.tools.ant.Project.executeTarget(Project.java:1298)
[teamFetch] at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
[teamFetch] at org.apache.tools.ant.Project.executeTargets(Project.java:1181)
[teamFetch] at org.apache.tools.ant.Main.runBuild(Main.java:698)
[teamFetch] at org.apache.tools.ant.Main.startAnt(Main.java:199)
[teamFetch] at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
[teamFetch] at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
[teamFetch] Caused by: java.net.SocketTimeoutException
[teamFetch] at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:250)
[teamFetch] at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
[teamFetch] at com.ibm.team.repository.transport.client.InterruptableSocket$InterruptableSocketInputStream.read(InterruptableSocket.java:583)
[teamFetch] at com.ibm.jsse2.a.a(a.java:167)
[teamFetch] at com.ibm.jsse2.a.a(a.java:64)
[teamFetch] at com.ibm.jsse2.jc.a(jc.java:130)
[teamFetch] at com.ibm.jsse2.jc.a(jc.java:551)
[teamFetch] at com.ibm.jsse2.e.read(e.java:38)
[teamFetch] at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
[teamFetch] at java.io.BufferedInputStream.read(BufferedInputStream.java:246)
[teamFetch] at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
[teamFetch] at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
[teamFetch] at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115)
[teamFetch] at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1373)
[teamFetch] at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832)
[teamFetch] at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
[teamFetch] at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
[teamFetch] at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
[teamFetch] at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
[teamFetch] at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
[teamFetch] at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
[teamFetch] at com.ibm.team.repository.transport.client.ClientHttpUtil.executeHttpMethod(ClientHttpUtil.java:148)
[teamFetch] ... 53 more
At that time there were network intrusion filters that were being triggered
on false positives and for a time the symptom above abated. Now the last
two attempts have failed. The customer has scheduled the build to start
at midnight and I have observed (via nmon) that on 23rd the I/O was very
high starting just after midnight and for about 1/2 hr after.
The server (as far as I'm aware) is tuned quite well and manages to provide service for 12 RTC repositories of varying size and activity.
I scanned the db2diag for last 1 day for the database in question and no
Error or Severe conditions were reported. No other system issues (e.g
full filesystem) have occurred in weeks.
The build toolkit is run under this version of Java:
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pap64dev-20080315 (SR7))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 AIX ppc64-64 j9vmap6423-20080315 (JIT enabled)
J9VM - 20080314_17962_BHdSMr
JIT - 20080130_0718ifx2_r8
GC - 200802_08)
JCL - 20080314
Server:
Installed Products
Jazz Foundation Server 1.0 (I20090618-1753)
Rational Team Concert 2.0 (I20090618-1753)
Server VM
J2RE 1.5.0 IBM J9 2.3 AIX ppc64-64 j9vmap6423ifx-20090422 (JIT enabled)
J9VM - 20090421_34076_BHdSMr
JIT - 20081112_1511ifx6_r8
GC - 200811_07
Any suggestions ?