[RTC Build] Fetching performance issue
We have a weird behavior with our build...
The Repository Workspace associated to the Build Definition contains 350,000 files.
When the build starts from scratch, fetching files in the build machine takes an expected time based on the footprint of our Workspace.
Now, if we change only 1 file in our associated Stream and we keep unchecked "Delete directory before loading' option in the build definition is checked?", it takes 2 hours "just" for fetching 1 single file...
We are running RTC 4.0.1... And, for our tests, the build engine is on the machine as the JTS.
We found few posts mentioning some similar issues:
- Jazz Build 'Fetching files' activity, performance question - Jazz Forum
- Build stuck at "75% (Fetching files)" - Jazz Forum
- Fetching to an existing source tree - Jazz Forum
Now we are not sure what we should do to fix that.
Any help will be welcome
Thanks a lot.
3 answers
Comments
Hi there,
We do not have have multiple build engines honoring this build definition. And the time given in the first message are worse. We have about 350 000 files in deferents components (about 150).
Here the first activity time for Fetching files (first build) :
-> Fetching files 1 h 13 m 22 s
The second activity time for Fetching files (second build)
-> Fetching files 7 h 59 m 15 s
Between the first and second build we changed nothing in the repository workspace (no change). The build engine and the RTC server are on the same machine.
OS :
- Windows Sever 2008 64 bits 64 G/Ram 64 CPU Intel Xeon
RTC
- Windows 64 bits 8 G/Ram
We understand the fteching time for the first build but not for the second build there are not change to load.
Regards,
Have you considered to put the JBE on its own machine?
Are there any timeouts in any of the log files?
What does performance monitoring (OS, JazzMon) show? Are you starting to swap?
What AppServer and what Database are you using?
Is the DB located on the same machine that has the JBE, the Jazz Server and Application Server?
Have you considered to put the JBE on its own machine?
-> Will try ASAP.
Are there any timeouts in any of the log files?
-> Nothing
What does performance monitoring (OS, JazzMon) show? Are you starting to swap?
-> No the process is just very long
What AppServer and what Database are you using?
-> Tomcat derby (default install)
Is the DB located on the same machine that has the JBE, the Jazz Server and Application Server?
-> Yes will first try to isolate the jbe.
If you use Derby for this kind of huge stream, you should expect issues. Derby is only barely thought to be able to handle small teams - and this includes small SCM sizes.
Derby is all working on the filesystem, you are probably bringing down your server with open handles or the disk glowing.
Migrate to DB2 and have DB2 on a dedicated DB Server.
As mentioned, there is a LOT of work that goes on to figure out what to load again.
many of our builds just load it all. every time. cause it easy.
the RTC file naming convention works very well with the squid caching proxy (or similar)
Comments
Hi,
I do not have time to test with db2. But I want to make a test with caching proxy. Do you have links to help me installing a caching proxy ?
Regards,
Please note that you can search Jazz.net.
https://jazz.net/library/article/325
Please note that I don't see why the performance should be better, if you have everything on the same machine. The JBE's and the caching proxy should be on another machine than the servers.
I manage to make all tests but for now I do not have enough time to do so. My intention is to install DB2 database , RTC and the jbe on separates machines.
- DB2 64 bits on Xeon (Windows 2008 R2 64 bits, 64 G Ram).
- RTC 64 bits on W530 (Windows Seven 64 bits, 16 G Ram).
-
Build Engine 32 bits on W520 (Windows Seven 64 bits, 16 G Ram)
- Fetching files 00:00:57 1 h 33 m 27 s
- Fetching files 00:00:11 9 h 28 m 42 s
- Fetching files 00:00:11 9 h 19 m 20 s
-
Debian installed in a VmWare installed on the W530
- Fetching files 00:00:57 1 h 09 m 12 s
- Fetching files 00:00:48 5 m 45 s
- Fetching files 00:00:10 5 m 41 s
So the problem is with Windows build engine code. I already mentioned this kind of differences between Windows and Linux in this work items. But it skill opened.
Long Time spends for "Loading" during sparse loading of large amount of files (208769)
Comments
Consider writing a work item here and provide your data.
Here the WI I created :
Long time during the fetching time of a build with large amount of files under Windows (258751)
Comments
Rob Logie
Mar 20 '13, 11:59 a.m.How does the RTC SCM system know what has changed in the local sandbox vis what needs to be downloaded fresh ?
Is it doing something like comparing all files in the repository workspace with the sandbox to find out what has changed ?
Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Mar 20 '13, 9:42 a.m.Philippe,
Spencer Murata
FORUM MODERATOR / JAZZ DEVELOPER Mar 20 '13, 3:13 p.m.To reduce the number of moving parts you might try to load the workspace outside of the build using the scm command line. If it also take a long time then we can throw out Build as being involved.