It's all about the answers!

Ask a question

Timeout on teamFetch operation


Kevin Ramer (4.5k9186201) | asked Jan 24 '10, 10:23 a.m.
We leverage the ant build toolkit to provide RTC users the access to our
in-house build infrastructure. A couple weeks ago a certain build began
encountering this:

[teamFetch] Not using a proxy to reach https://rtp-rtc10:9443/jazz

[teamFetch] Fetching contents of workspace "tbsm_dev Production Build Workspace" to "/bigFS/rtp/FS3/tbsm_dev/prod/TOP".
[teamFetch] com.ibm.team.repository.common.transport.TeamServiceException: CRJAZ0099I HTTP error "java.net.SocketTimeoutException" accessing URL "https://rtp-rtc10:9443/jazz/service/com.ibm.team.filesystem.common.IFilesystemService".
[teamFetch] at com.ibm.team.repository.transport.client.ClientHttpUtil.executeHttpMethod(ClientHttpUtil.java:179)
[teamFetch] at com.ibm.team.repository.transport.client.RemoteTeamService.executeCancelableHttpMethod(RemoteTeamService.java:554)
[teamFetch] at com.ibm.team.repository.transport.client.RemoteTeamService.invokePost(RemoteTeamService.java:527)
[teamFetch] at com.ibm.team.repository.transport.client.RemoteTeamService.executeMethod(RemoteTeamService.java:467)
[teamFetch] at com.ibm.team.repository.transport.client.RemoteTeamService.invoke(RemoteTeamService.java:197)
[teamFetch] at com.ibm.team.repository.transport.client.ServiceInvocationHandler.invoke(ServiceInvocationHandler.java:43)
[teamFetch] at $Proxy23.getFileTreeByVersionable(Unknown Source)
[teamFetch] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[teamFetch] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
[teamFetch] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[teamFetch] at java.lang.reflect.Method.invoke(Method.java:618)
[teamFetch] at com.ibm.team.repository.client.internal.ServiceInterfaceProxy.invokeServiceCall(ServiceInterfaceProxy.java:149)
[teamFetch] at com.ibm.team.repository.client.internal.ServiceInterfaceProxy.invoke(ServiceInterfaceProxy.java:84)
[teamFetch] at $Proxy23.getFileTreeByVersionable(Unknown Source)
[teamFetch] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[teamFetch] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
[teamFetch] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[teamFetch] at java.lang.reflect.Method.invoke(Method.java:618)
[teamFetch] at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy$2.run(ScmServiceInterfaceProxy.java:164)
[teamFetch] at com.ibm.team.repository.client.internal.TeamRepository$3.run(TeamRepository.java:1164)
[teamFetch] at com.ibm.team.repository.common.transport.CancelableCaller.call(CancelableCaller.java:79)
[teamFetch] at com.ibm.team.repository.client.internal.TeamRepository.callCancelableService(TeamRepository.java:1157)
[teamFetch] at com.ibm.team.scm.client.internal.ScmClientLibraryContext.callCancelableService(ScmClientLibraryContext.java:70)
[teamFetch] at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy.invokeCancelableService(ScmServiceInterfaceProxy.java:161)
[teamFetch] at com.ibm.team.scm.client.internal.ScmServiceInterfaceProxy.invoke(ScmServiceInterfaceProxy.java:91)
[teamFetch] at $Proxy23.getFileTreeByVersionable(Unknown Source)
[teamFetch] at com.ibm.team.filesystem.client.internal.FileSystemServiceProxy.getFileTreeByVersionable(FileSystemServiceProxy.java:262)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.LoadOperation.load(LoadOperation.java:1053)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.LoadOperation$2.run(LoadOperation.java:689)
[teamFetch] at com.ibm.team.filesystem.client.internal.SharingManager.runWithinFileSystemLock(SharingManager.java:730)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.FileSystemOperation.runWithinFileSystemLock(FileSystemOperation.java:173)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.FileSystemOperation.runWithinFileSystemLock(FileSystemOperation.java:153)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.LoadOperation.execute(LoadOperation.java:721)
[teamFetch] at com.ibm.team.filesystem.client.internal.operations.FileSystemOperation.run(FileSystemOperation.java:90)
[teamFetch] at com.ibm.team.build.internal.scm.SourceControlUtility.updateFileCopyArea(SourceControlUtility.java:364)
[teamFetch] at com.ibm.team.build.internal.scm.SourceControlUtility.updateFileCopyArea(SourceControlUtility.java:175)
[teamFetch] at com.ibm.team.build.ant.task.TeamFetchTask.doExecute(TeamFetchTask.java:124)
[teamFetch] at com.ibm.team.build.ant.task.AbstractTeamBuildTask.execute(AbstractTeamBuildTask.java:433)
[teamFetch] at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
[teamFetch] at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
[teamFetch] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[teamFetch] at java.lang.reflect.Method.invoke(Method.java:618)
[teamFetch] at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:105)
[teamFetch] at org.apache.tools.ant.Task.perform(Task.java:348)
[teamFetch] at org.apache.tools.ant.Target.execute(Target.java:357)
[teamFetch] at org.apache.tools.ant.Target.performTasks(Target.java:385)
[teamFetch] at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1329)
[teamFetch] at org.apache.tools.ant.Project.executeTarget(Project.java:1298)
[teamFetch] at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
[teamFetch] at org.apache.tools.ant.Project.executeTargets(Project.java:1181)
[teamFetch] at org.apache.tools.ant.Main.runBuild(Main.java:698)
[teamFetch] at org.apache.tools.ant.Main.startAnt(Main.java:199)
[teamFetch] at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
[teamFetch] at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
[teamFetch] Caused by: java.net.SocketTimeoutException
[teamFetch] at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:250)
[teamFetch] at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
[teamFetch] at com.ibm.team.repository.transport.client.InterruptableSocket$InterruptableSocketInputStream.read(InterruptableSocket.java:583)
[teamFetch] at com.ibm.jsse2.a.a(a.java:167)
[teamFetch] at com.ibm.jsse2.a.a(a.java:64)
[teamFetch] at com.ibm.jsse2.jc.a(jc.java:130)
[teamFetch] at com.ibm.jsse2.jc.a(jc.java:551)
[teamFetch] at com.ibm.jsse2.e.read(e.java:38)
[teamFetch] at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
[teamFetch] at java.io.BufferedInputStream.read(BufferedInputStream.java:246)
[teamFetch] at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
[teamFetch] at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
[teamFetch] at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115)
[teamFetch] at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1373)
[teamFetch] at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832)
[teamFetch] at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
[teamFetch] at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
[teamFetch] at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
[teamFetch] at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
[teamFetch] at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
[teamFetch] at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
[teamFetch] at com.ibm.team.repository.transport.client.ClientHttpUtil.executeHttpMethod(ClientHttpUtil.java:148)
[teamFetch] ... 53 more



At that time there were network intrusion filters that were being triggered
on false positives and for a time the symptom above abated. Now the last
two attempts have failed. The customer has scheduled the build to start
at midnight and I have observed (via nmon) that on 23rd the I/O was very
high starting just after midnight and for about 1/2 hr after.

The server (as far as I'm aware) is tuned quite well and manages to provide service for 12 RTC repositories of varying size and activity.

I scanned the db2diag for last 1 day for the database in question and no
Error or Severe conditions were reported. No other system issues (e.g
full filesystem) have occurred in weeks.

The build toolkit is run under this version of Java:

java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pap64dev-20080315 (SR7))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 AIX ppc64-64 j9vmap6423-20080315 (JIT enabled)
J9VM - 20080314_17962_BHdSMr
JIT - 20080130_0718ifx2_r8
GC - 200802_08)
JCL - 20080314

Server:
Installed Products
Jazz Foundation Server 1.0 (I20090618-1753)
Rational Team Concert 2.0 (I20090618-1753)

Server VM
J2RE 1.5.0 IBM J9 2.3 AIX ppc64-64 j9vmap6423ifx-20090422 (JIT enabled)
J9VM - 20090421_34076_BHdSMr
JIT - 20081112_1511ifx6_r8
GC - 200811_07



Any suggestions ?

4 answers



permanent link
Kevin Ramer (4.5k9186201) | answered Jan 26 '10, 8:29 a.m.
I searched work items and found mention of setting a timeout property in preference.properties. Looking at the RTC Documentation searching for 'preferences' lead me to this:


preferences.properties file
Properties for the scm command line client can be controlled through the preferences.properties file. By default, the file resides in ~/.jazz-scm, but other locations can be specified by setting scm config /path/to/other/location. The file can contain two values: repository.timeout and password.mask.
repository.timeout

This value controls the duration of the connection timeout between the scm executable and the Jazz repository. It is set in seconds.
password.mask

When set to true then the user's password is hidden on password prompts (when -P is not specified on the command line). If it is set to any other value, the user's password is echoed.

permanent link
Kevin Ramer (4.5k9186201) | answered Jan 27 '10, 6:50 a.m.
Well, this helped ... ONCE.

Guess it's time to haunt the Work Items.

permanent link
Evan Hughes (2.4k1318) | answered Jan 27 '10, 9:50 a.m.
JAZZ DEVELOPER
The preference.properties change won't modify the behaviour of the ant load tasks.

permanent link
Nick Edgar (6.5k711) | answered Jan 28 '10, 11:15 a.m.
JAZZ DEVELOPER
Let's follow up in the workitem you created: 97960: SocketTimeoutException during build

Your answer


Register or to post your answer.


Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.