How to fetch file faster
We have a large project with thousands of source files in RTC (v4.0). It take more than 50 minutes to fetch the changed files (even there is only one file changed since last build). Is there a way we can make the file fetching faster - we tried to set up an incremental build that is scheduled to run every 10 minutes.
I noticed there is a build property "team.scm.maxContentThreads". Is this property supported in V4.0? If so, will it help if I set the value to a large number, for example, "team.scm.maxContentThreads=100"? |
2 answers
It shouldn't take that long to fetch if you are reusing the workspace. Our recommendation is to have a dedicated build workspace on the build machine and fetch the files into that workspace everytime and reserve a 'clean' workspace to just when you need it. Especially so if you are doing incremental continuous integration builds as it will be lots of builds. If you fetch into the existing workspace it will only to the changed files like you expected.
~Spencer
Comments Thanks Spencer for the quick answer. We do have the dedicated workspace for the build.
Interesting. The slow performance then is during the compare of the snapshots and not the actual fetch. I don't know how to optimize the compare besides limiting the load components or breaking the streams down more granularly. Sorry Spencer, why do you think the time is spent on the snapshot comparison? It seems that "accepting changes" only took few seconds and 45 minutes are on the "fetching". BTW, the source code is 4+ GB with 250K+ files. Is there a way I can let RTC not to load the code for me, instead, I will load the changed code with SCM command (but I still want the build be triggered by the code change though)?
Spencer Murata
commented May 14 '15, 11:48 a.m.
| edited May 14 '15, 11:49 a.m.
FORUM MODERATOR / JAZZ DEVELOPER
The compare is lumped into the fetch since we have to determine what files have changed between the two snapshots and if there are a lot of files it can take a while. You could make it not load by making a load rule that indicates to not load anything, and then put the SCM command into the build batch script or whatever you are running. But with so many files it might be work thinking about splitting it into more components and breaking the build up. Thanks Spencer, I will try to play around with the load rule and SCM command to see if it works. It is not possible to break the streams/components - this is actually for an IBM product, Curam. It is now built in CruiseControl (with RTC SCM CLI) and we are trying to bring it to Jazz build. Everything works except the load/fetch issue.
Do you have an example of the load rule to load nothing from the component. I have tried the following:
<parentLoadRule>
<component name="MyComponent" />
<parentFolder repositoryPath="client" />
<exclude>
<filter pattern="*"/>
</exclude>
</parentLoadRule>
And I got the error like this:
com.ibm.team.filesystem.client.FileSystemStatusException: Status INFO: com.ibm.team.filesystem.client code=0 There are no qualifying items under /client in component MyComponent to be loaded into sandbox D:\Build_LoadDir\null
showing 5 of 6
show 1 more comments
|
when we had this problem back in the 3.x days, I designed a squid proxy server infrastructure to offload the downloading to get the best performance possible
see https://jazz.net/forum/questions/112889/caching-server-useful-for-rtc-workitems |