Jazz Forum Welcome to the Jazz Community Forum Connect and collaborate with IBM Engineering experts and users

How to fetch file faster

We have a large project with thousands of source files in RTC (v4.0). It take more than 50 minutes to fetch the changed files (even there is only one file changed since last build). Is there a way we can make the file fetching faster - we tried to set up an incremental build that is scheduled to run every 10 minutes.

I noticed there is a build property "team.scm.maxContentThreads". Is this property supported in V4.0? If so, will it help if I set the value to a large number, for example, "team.scm.maxContentThreads=100"?

0 votes



2 answers

Permanent link
 It shouldn't take that long to fetch if you are reusing the workspace.  Our recommendation is to have a dedicated build workspace on the build machine and fetch the files into that workspace everytime and reserve a 'clean' workspace to just when you need it.  Especially so if you are doing incremental continuous integration builds as it will be lots of builds.  If you fetch into the existing workspace it will only to the changed files like you expected.

~Spencer

1 vote

Comments

Thanks Spencer for the quick answer. We do have the dedicated workspace for the build.
I am attaching some log information below. You can see there is only one change <found changes (1)>, but the fetching took 45 minutes (08:12:29  - 08:59:12). From the load folder on the build server, I can see that only the changed files was reloaded. Any idea what the time is spent on and how to resolve it?
...
2015-05-14 08:12:23 [Jazz build engine] Checking for changes incoming to workspace "Curam Vendor Stream Workspace - RTCBuild" ...
2015-05-14 08:12:23 [Jazz build engine]  found changes (1)
...
2015-05-14 08:12:24 [Jazz build engine] Accepting changes into workspace "Curam Vendor Stream Workspace - RTCBuild" ...
2015-05-14 08:12:29 [Jazz build engine] Fetching files to fetch destination "D:\Build_LoadDir\SAMS_OOTB_Incremental" ...
2015-05-14 08:59:12
[Jazz build engine] Invoking build participant "com.ibm.team.build.ant"

Interesting.  The slow performance then is during the compare of the snapshots and not the actual fetch.  I don't know how to optimize the compare besides limiting the load components or breaking the streams down more granularly.

Sorry Spencer, why do you think the time is spent on the snapshot comparison? It seems that "accepting changes" only took few seconds and 45 minutes are on the "fetching". BTW, the source code is 4+ GB with 250K+ files. Is there a way I can let RTC not to load the code for me, instead, I will load the changed code with SCM command (but I still want the build be triggered by the code change though)?

 The compare is lumped into the fetch since we have to determine what files have changed between the two snapshots and if there are a lot of files it can take a while.  You could make it not load by making a load rule that indicates to not load anything, and then put the SCM command into the build batch script or whatever you are running.  But with so many files it might be work thinking about splitting it into more components and breaking the build up.

Thanks Spencer, I will try to play around with the load rule and SCM command to see if it works. It is not possible to break the streams/components - this is actually for an IBM product, Curam. It is now built in CruiseControl (with RTC SCM CLI) and we are trying to bring it to Jazz build. Everything works except the load/fetch issue.

Do you have an example of the load rule to load nothing from the component. I have tried the following: 

    <parentLoadRule>
        <component name="MyComponent" />
        <parentFolder repositoryPath="client" />
        <exclude>
            <filter pattern="*"/>
        </exclude>
    </parentLoadRule>

And I got the error like this:
com.ibm.team.filesystem.client.FileSystemStatusException: Status INFO: com.ibm.team.filesystem.client code=0 There are no qualifying items under /client in component MyComponent to be loaded into sandbox D:\Build_LoadDir\null

showing 5 of 6 show 1 more comments

Permanent link
when we had this problem back in the 3.x days, I designed a squid proxy server infrastructure to offload the downloading to get  the best performance possible

see
https://jazz.net/forum/questions/112889/caching-server-useful-for-rtc-workitems

0 votes

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Search context
Follow this question

By Email: 

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers
Answers and Comments
Question details
× 12,014
× 562

Question asked: May 14 '15, 10:00 a.m.

Question was seen: 4,181 times

Last updated: Jun 01 '15, 3:35 p.m.

Confirmation Cancel Confirm