[RTC Build] Fetching performance issue

Philippe Krief (561●1●5●5) | asked Mar 19 '13, 7:09 p.m.
JAZZ DEVELOPER
edited Mar 20 '13, 3:13 p.m. by Spencer Murata (2.3k●11●59●71)

Hi Folks

We have a weird behavior with our build...

The Repository Workspace associated to the Build Definition contains 350,000 files.
When the build starts from scratch, fetching files in the build machine takes an expected time based on the footprint of our Workspace.
Now, if we change only 1 file in our associated Stream and we keep unchecked "Delete directory before loading' option in the build definition is checked?", it takes 2 hours "just" for fetching 1 single file...

We are running RTC 4.0.1... And, for our tests, the build engine is on the machine as the JTS.

We found few posts mentioning some similar issues:

In my understanding, this performance issue can be explained when I read the first post...
Now we are not sure what we should do to fix that.
Any help will be welcome

Thanks a lot.

Comments

Rob Logie commented Mar 19 '13, 9:04 p.m. | edited Mar 20 '13, 11:59 a.m.

How does the RTC SCM system know what has changed in the local sandbox vis what needs to be downloaded fresh ?

Is it doing something like comparing all files in the repository workspace with the sandbox to find out what has changed ?

Ralph Schoon commented Mar 20 '13, 9:42 a.m.

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Philippe,

how long does the first fetch take?
I assume you build always in the same folder, correct? If you don't the "Delete before..." does not matter.
How many components do you have and how many files per component (average, thumbs number)
Is the Server remote or local?
If the server is remote, have you considered a Squid caching proxy local or close to the build engine?

Spencer Murata commented Mar 20 '13, 3:13 p.m.

FORUM MODERATOR / JAZZ DEVELOPER

To reduce the number of moving parts you might try to load the workspace outside of the build using the scm command line. If it also take a long time then we can throw out Build as being involved.

~Spencer

3 answers

Most liked answers ↑|Newest answers|Oldest answers

permanent link

John Camelon (1.7k●1●4) | answered Mar 21 '13, 8:43 p.m.
JAZZ DEVELOPER

The build activities should indicate how much of the activity is spent doing the load vs. the other pieces.

Reusing a build workspace and not deleting all the files every time will reduce the amount of content that gets transferred to the build machine, but does not make the cost of "load" free. We still need to figure out what the current configuration is, and figure out what delta needs to be downloaded from the client.

Additionally, if you have multiple build engines honouring this build definition, you may find that the file areas get "out of sync" if build engine 1 last ran the build, and build engine 2's repository workspace is out of date.

Are either of these conditions happening for you?

Comments

Jean-Yves Baudy commented Mar 22 '13, 2:33 a.m.

Hi there,
We do not have have multiple build engines honoring this build definition. And the time given in the first message are worse. We have about 350 000 files in deferents components (about 150).
Here the first activity time for Fetching files (first build) :
-> Fetching files 1 h 13 m 22 s
The second activity time for Fetching files (second build)
-> Fetching files 7 h 59 m 15 s
Between the first and second build we changed nothing in the repository workspace (no change). The build engine and the RTC server are on the same machine.
OS :
- Windows Sever 2008 64 bits 64 G/Ram 64 CPU Intel Xeon
RTC
- Windows 64 bits 8 G/Ram

We understand the fteching time for the first build but not for the second build there are not change to load.
Regards,

Ralph Schoon commented Mar 22 '13, 4:33 a.m.

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Have you considered to put the JBE on its own machine?
Are there any timeouts in any of the log files?
What does performance monitoring (OS, JazzMon) show? Are you starting to swap?
What AppServer and what Database are you using?
Is the DB located on the same machine that has the JBE, the Jazz Server and Application Server?

Jean-Yves Baudy commented Mar 22 '13, 6:53 a.m.

Have you considered to put the JBE on its own machine?
-> Will try ASAP.
Are there any timeouts in any of the log files?
-> Nothing
What does performance monitoring (OS, JazzMon) show? Are you starting to swap?
-> No the process is just very long
What AppServer and what Database are you using?
-> Tomcat derby (default install)
Is the DB located on the same machine that has the JBE, the Jazz Server and Application Server?
-> Yes will first try to isolate the jbe.

Ralph Schoon commented Mar 22 '13, 6:59 a.m.

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

If you use Derby for this kind of huge stream, you should expect issues. Derby is only barely thought to be able to handle small teams - and this includes small SCM sizes.

Derby is all working on the filesystem, you are probably bringing down your server with open handles or the disk glowing.

Migrate to DB2 and have DB2 on a dedicated DB Server.

permanent link

sam detweiler (12.5k●6●194●201) | answered Mar 21 '13, 10:39 p.m.

we are adding caching proxies to our environment to solve this problem.
As mentioned, there is a LOT of work that goes on to figure out what to load again.
many of our builds just load it all. every time. cause it easy.

the RTC file naming convention works very well with the squid caching proxy (or similar)

Comments

Jean-Yves Baudy commented Mar 25 '13, 5:38 a.m.

Hi,
I do not have time to test with db2. But I want to make a test with caching proxy. Do you have links to help me installing a caching proxy ?
Regards,

Ralph Schoon commented Mar 25 '13, 6:11 a.m.

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Please note that you can search Jazz.net.

https://jazz.net/library/article/325

Ralph Schoon commented Mar 25 '13, 6:13 a.m.

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Please note that I don't see why the performance should be better, if you have everything on the same machine. The JBE's and the caching proxy should be on another machine than the servers.

Jean-Yves Baudy commented Mar 25 '13, 9:37 a.m.

I manage to make all tests but for now I do not have enough time to do so. My intention is to install DB2 database , RTC and the jbe on separates machines.

permanent link

Jean-Yves Baudy (3●1●2) | answered Apr 02 '13, 11:26 a.m.

Here the results with this configuration

DB2 64 bits on Xeon (Windows 2008 R2 64 bits, 64 G Ram).
RTC 64 bits on W530 (Windows Seven 64 bits, 16 G Ram).
Build Engine 32 bits on W520 (Windows Seven 64 bits, 16 G Ram)

First build :

Fetching files 00:00:57 1 h 33 m 27 s

Second build (no changes in the loaded stream) :

Fetching files 00:00:11 9 h 28 m 42 s

Third build (no changes in the loaded stream) :

Fetching files 00:00:11 9 h 19 m 20 s

A made another test where I used Linux for the build engine.

Debian installed in a VmWare installed on the W530

First build

Fetching files 00:00:57 1 h 09 m 12 s

Second build (no changes in the loaded stream) :

Fetching files 00:00:48 5 m 45 s

Third build (no changes in the loaded stream) :

Fetching files 00:00:10 5 m 41 s

So the problem is with Windows build engine code. I already mentioned this kind of differences between Windows and Linux in this work items. But it skill opened.

Long Time spends for "Loading" during sparse loading of large amount of files (208769)

Comments

Ralph Schoon commented Apr 02 '13, 11:46 a.m. | edited Apr 02 '13, 11:46 a.m.

FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER

Consider writing a work item here and provide your data.

Jean-Yves Baudy commented Apr 02 '13, 11:58 a.m.

Here the WI I created :
Long time during the fetching time of a build with large amount of files under Windows (258751)

It's all about the answers!

Related questions

[RTC Build] Fetching performance issue

3 answers

Your answer