How to fetch file faster
I noticed there is a build property "team.scm.maxContentThreads". Is this property supported in V4.0? If so, will it help if I set the value to a large number, for example, "team.scm.maxContentThreads=100"?
2 answers
Comments
Thanks Spencer for the quick answer. We do have the dedicated workspace for the build.
I am attaching some log information below. You can see there is only one change <found changes (1)>, but the fetching took 45 minutes (08:12:29 - 08:59:12). From the load folder on the build server, I can see that only the changed files was reloaded. Any idea what the time is spent on and how to resolve it?
...
2015-05-14 08:12:23 [Jazz build engine] Checking for changes incoming to workspace "Curam Vendor Stream Workspace - RTCBuild" ...
2015-05-14 08:12:23 [Jazz build engine] found changes (1)
...
2015-05-14 08:12:24 [Jazz build engine] Accepting changes into workspace "Curam Vendor Stream Workspace - RTCBuild" ...
2015-05-14 08:12:29 [Jazz build engine] Fetching files to fetch destination "D:\Build_LoadDir\SAMS_OOTB_Incremental" ...
2015-05-14 08:59:12 [Jazz build engine] Invoking build participant "com.ibm.team.build.ant"
Interesting. The slow performance then is during the compare of the snapshots and not the actual fetch. I don't know how to optimize the compare besides limiting the load components or breaking the streams down more granularly.
Sorry Spencer, why do you think the time is spent on the snapshot comparison? It seems that "accepting changes" only took few seconds and 45 minutes are on the "fetching". BTW, the source code is 4+ GB with 250K+ files. Is there a way I can let RTC not to load the code for me, instead, I will load the changed code with SCM command (but I still want the build be triggered by the code change though)?
The compare is lumped into the fetch since we have to determine what files have changed between the two snapshots and if there are a lot of files it can take a while. You could make it not load by making a load rule that indicates to not load anything, and then put the SCM command into the build batch script or whatever you are running. But with so many files it might be work thinking about splitting it into more components and breaking the build up.
Thanks Spencer, I will try to play around with the load rule and SCM command to see if it works. It is not possible to break the streams/components - this is actually for an IBM product, Curam. It is now built in CruiseControl (with RTC SCM CLI) and we are trying to bring it to Jazz build. Everything works except the load/fetch issue.
Do you have an example of the load rule to load nothing from the component. I have tried the following: