It's all about the answers!

Ask a question

How to fetch file faster

Kevin Lou (311925) | asked May 14 '15, 10:00 a.m.
We have a large project with thousands of source files in RTC (v4.0). It take more than 50 minutes to fetch the changed files (even there is only one file changed since last build). Is there a way we can make the file fetching faster - we tried to set up an incremental build that is scheduled to run every 10 minutes.

I noticed there is a build property "team.scm.maxContentThreads". Is this property supported in V4.0? If so, will it help if I set the value to a large number, for example, "team.scm.maxContentThreads=100"?

2 answers

permanent link
Spencer Murata (2.3k115971) | answered May 14 '15, 10:55 a.m.
 It shouldn't take that long to fetch if you are reusing the workspace.  Our recommendation is to have a dedicated build workspace on the build machine and fetch the files into that workspace everytime and reserve a 'clean' workspace to just when you need it.  Especially so if you are doing incremental continuous integration builds as it will be lots of builds.  If you fetch into the existing workspace it will only to the changed files like you expected.


Kevin Lou commented May 14 '15, 11:14 a.m.

Thanks Spencer for the quick answer. We do have the dedicated workspace for the build.
I am attaching some log information below. You can see there is only one change <found changes (1)>, but the fetching took 45 minutes (08:12:29  - 08:59:12). From the load folder on the build server, I can see that only the changed files was reloaded. Any idea what the time is spent on and how to resolve it?
2015-05-14 08:12:23 [Jazz build engine] Checking for changes incoming to workspace "Curam Vendor Stream Workspace - RTCBuild" ...
2015-05-14 08:12:23 [Jazz build engine]  found changes (1)
2015-05-14 08:12:24 [Jazz build engine] Accepting changes into workspace "Curam Vendor Stream Workspace - RTCBuild" ...
2015-05-14 08:12:29 [Jazz build engine] Fetching files to fetch destination "D:\Build_LoadDir\SAMS_OOTB_Incremental" ...
2015-05-14 08:59:12
[Jazz build engine] Invoking build participant ""

Spencer Murata commented May 14 '15, 11:20 a.m.

Interesting.  The slow performance then is during the compare of the snapshots and not the actual fetch.  I don't know how to optimize the compare besides limiting the load components or breaking the streams down more granularly.

Kevin Lou commented May 14 '15, 11:35 a.m.

Sorry Spencer, why do you think the time is spent on the snapshot comparison? It seems that "accepting changes" only took few seconds and 45 minutes are on the "fetching". BTW, the source code is 4+ GB with 250K+ files. Is there a way I can let RTC not to load the code for me, instead, I will load the changed code with SCM command (but I still want the build be triggered by the code change though)?

Spencer Murata commented May 14 '15, 11:48 a.m. | edited May 14 '15, 11:49 a.m.

 The compare is lumped into the fetch since we have to determine what files have changed between the two snapshots and if there are a lot of files it can take a while.  You could make it not load by making a load rule that indicates to not load anything, and then put the SCM command into the build batch script or whatever you are running.  But with so many files it might be work thinking about splitting it into more components and breaking the build up.

Kevin Lou commented May 14 '15, 12:00 p.m.

Thanks Spencer, I will try to play around with the load rule and SCM command to see if it works. It is not possible to break the streams/components - this is actually for an IBM product, Curam. It is now built in CruiseControl (with RTC SCM CLI) and we are trying to bring it to Jazz build. Everything works except the load/fetch issue.

Kevin Lou commented Jun 01 '15, 8:19 a.m.

Do you have an example of the load rule to load nothing from the component. I have tried the following: 

        <component name="MyComponent" />
        <parentFolder repositoryPath="client" />
            <filter pattern="*"/>

And I got the error like this: Status INFO: code=0 There are no qualifying items under /client in component MyComponent to be loaded into sandbox D:\Build_LoadDir\null

showing 5 of 6 show 1 more comments

permanent link
sam detweiler (12.5k6194201) | answered Jun 01 '15, 3:35 p.m.
when we had this problem back in the 3.x days, I designed a squid proxy server infrastructure to offload the downloading to get  the best performance possible


Your answer

Register or to post your answer.