How to fetch file faster

Kevin Lou

(31●21●25) May 14 '15, 10:00 a.m.

We have a large project with thousands of source files in RTC (v4.0). It take more than 50 minutes to fetch the changed files (even there is only one file changed since last build). Is there a way we can make the file fetching faster - we tried to set up an incremental build that is scheduled to run every 10 minutes.

I noticed there is a build property "team.scm.maxContentThreads". Is this property supported in V4.0? If so, will it help if I set the value to a large number, for example, "team.scm.maxContentThreads=100"?

0 votes

2 answers

6,722 views

0 votes

2 answers

Permanent link

Spencer Murata

FORUM MODERATOR / JAZZ DEVELOPER (2.3k●11●59●71) May 14 '15, 10:55 a.m.

It shouldn't take that long to fetch if you are reusing the workspace. Our recommendation is to have a dedicated build workspace on the build machine and fetch the files into that workspace everytime and reserve a 'clean' workspace to just when you need it. Especially so if you are doing incremental continuous integration builds as it will be lots of builds. If you fetch into the existing workspace it will only to the changed files like you expected.

~Spencer

1 vote

Comments

Kevin Lou

May 14 '15, 11:14 a.m.

Thanks Spencer for the quick answer. We do have the dedicated workspace for the build. I am attaching some log information below. You can see there is only one change <found changes (1)>, but the fetching took 45 minutes (08:12:29 - 08:59:12). From the load folder on the build server, I can see that only the changed files was reloaded. Any idea what the time is spent on and how to resolve it? ... 2015-05-14 08:12:23 [Jazz build engine] Checking for changes incoming to workspace "Curam Vendor Stream Workspace - RTCBuild" ... 2015-05-14 08:12:23 [Jazz build engine] found changes (1) ... 2015-05-14 08:12:24 [Jazz build engine] Accepting changes into workspace "Curam Vendor Stream Workspace - RTCBuild" ... 2015-05-14 08:12:29 [Jazz build engine] Fetching files to fetch destination "D:\Build_LoadDir\SAMS_OOTB_Incremental" ... 2015-05-14 08:59:12 [Jazz build engine] Invoking build participant "com.ibm.team.build.ant"

Spencer Murata

FORUM MODERATOR / JAZZ DEVELOPER May 14 '15, 11:20 a.m.

Interesting. The slow performance then is during the compare of the snapshots and not the actual fetch. I don't know how to optimize the compare besides limiting the load components or breaking the streams down more granularly.

Kevin Lou

May 14 '15, 11:35 a.m.

Sorry Spencer, why do you think the time is spent on the snapshot comparison? It seems that "accepting changes" only took few seconds and 45 minutes are on the "fetching". BTW, the source code is 4+ GB with 250K+ files. Is there a way I can let RTC not to load the code for me, instead, I will load the changed code with SCM command (but I still want the build be triggered by the code change though)?

Spencer Murata

FORUM MODERATOR / JAZZ DEVELOPER May 14 '15, 11:49 a.m.

The compare is lumped into the fetch since we have to determine what files have changed between the two snapshots and if there are a lot of files it can take a while. You could make it not load by making a load rule that indicates to not load anything, and then put the SCM command into the build batch script or whatever you are running. But with so many files it might be work thinking about splitting it into more components and breaking the build up.

Kevin Lou

May 14 '15, 12:00 p.m.

Thanks Spencer, I will try to play around with the load rule and SCM command to see if it works. It is not possible to break the streams/components - this is actually for an IBM product, Curam. It is now built in CruiseControl (with RTC SCM CLI) and we are trying to bring it to Jazz build. Everything works except the load/fetch issue.

Kevin Lou

Jun 01 '15, 8:19 a.m.

Do you have an example of the load rule to load nothing from the component. I have tried the following: <parentLoadRule> <component name="MyComponent" /> <parentFolder repositoryPath="client" /> <exclude> <filter pattern="*"/> </exclude> </parentLoadRule> And I got the error like this: com.ibm.team.filesystem.client.FileSystemStatusException: Status INFO: com.ibm.team.filesystem.client code=0 There are no qualifying items under /client in component MyComponent to be loaded into sandbox D:\Build_LoadDir\null

showing 5 of 6 show 1 more comments

Permanent link

sam detweiler

(12.5k●6●195●201) Jun 01 '15, 3:35 p.m.

when we had this problem back in the 3.x days, I designed a squid proxy server infrastructure to offload the downloading to get the best performance possible

see
https://jazz.net/forum/questions/112889/caching-server-useful-for-rtc-workitems

0 votes

Your answer

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers

Answers and Comments

Question details

rational-team-concert

× 43,076

usage

× 12,144

build

× 563

teamfetch

× 2

Question asked: May 14 '15, 10:00 a.m.

Question was seen: 6,722 times

Last updated: Jun 01 '15, 3:35 p.m.

How to fetch file faster

2 answers

6,722 views

0 votes

2 answers

Comments

Your answer

Follow this question

Question details

Related questions