poor CC connector performance

Hi,

once the files have been synced it should get way faster, since only files that change are re synced.

Ralph

Hi All,

just share with you, here is the performance test result for CC connector in our environment,

1) CC to RTC: about 20,000 files to sync, took about one day to do that (it took about half hour to import these 20,000 files from file system into the same RTC server with RTC client).
2) RTC to CC: about 20,000 files to sync, took more than one day (it took about 2 hour to import these 20,000 files from file system into the same CC server with clearfsimport).

the performance is not acceptable to us, so if you are thinking about using CC connector to do something, and you do have more than 20,000 files in your project, please test it first, and if you have a better performance with CC connector, please kindly let us know, maybe there are some tips ....

Regards,
Makson

Hi Ralph,

yes, you are right, less file to sync, less time it take, if you still have lot of files needed to sync, then the performance is still a problem, and if you say that CC Connector is just suitable for synchronizing files less than 20,000, okay, please just mark it as production limitation, we did not know this before we purchased it ....

Regards,
Makson

As Ralph correctly pointed out, the purpose of the CC synchronizer is to
keep a a stream in RTC synchronized with a stream/branch in ClearCase,
so it is designed to run the incremental nightly sync quickly, at the
cost of slow startup time.

For details on the expected performance, and guidance on how to do the
initial sync, please see http://jazz.net/library/techtip/68

For a workitem on speeding up the initial sync, please see workitem 114744.

Cheers,
Geoff

On 5/31/2010 11:23 AM, cdlee wrote:

Hi All,

just share with you, here is the performance test result for CC
connector in our environment,

1) CC to RTC: about 20,000 files to sync, took about one day to do
that (it took about half hour to import these 20,000 files from file
system into the same RTC server with RTC client).
2) RTC to CC: about 20,000 files to sync, took more than one day (it
took about 2 hour to import these 20,000 files from file system into
the same CC server with clearfsimport).

the performance is not acceptable to us, so if you are thinking about
using CC connector to do something, and you do have more than 20,000
files in your project, please test it first, and if you have a better
performance with CC connector, please kindly let us know, maybe there
are some tips ....

Regards,
Makson

Hi Ralph,

Think the scenario that we have several dev streams and one integration stream, one release stream. And release stream is the CC synchronization stream. The changes will be accumulated to a certain level then sync to the other side. Thousands files changed is frequently in real case.

Beside, the statement "CC Connector is just suitable for synchronizing files less than 20,000" cdlee mentioned is too optimistic. The sync speed of CC connector goes down as file increases. The project with less than 5,000 files may consider to use it.

Hi Joseph,

I am aware of the sync time for the initial sync and am not happy about it either. However, once that is done the issue goes away to some extent (assuming the number of files changed during the day is way smaller than the initial number). The technote mentioned by Geoff helped me with the same initial issue.

It is also possible to speed up the sync time to some extent by optimizing the synchronization host for clear case performance, I guess. I saw slides indicating the synchronizer is used in IBM so it is used and usable.

Also like Geoff pointed out you could write a work item and provide information about your situation requesting for optimization.
To be honest, I have not written the code and don't know what the bottlenecks are. My assumption is that the synchronizer does a complex work comparing files and history of files trying to determine differences in RTC and CC.

Ralph

Hi Ralph,

Think the scenario that we have several dev streams and one integration stream, one release stream. And release stream is the CC synchronization stream. The changes will be accumulated to a certain level then sync to the other side. Thousands files changed is frequently in real case.

Beside, the statement "CC Connector is just suitable for synchronizing files less than 20,000" cdlee mentioned is too optimistic. The sync speed of CC connector goes down as file increases. The project with less than 5,000 files may consider to use it.

The CC synchronization branch would normally be a child of the CC
integration branch(commonly a sibling of the CC development branches).
Similarly, the RTC synchronization stream would normally be a child of
the RTC integration stream. The integration stream should be merged into
the synchronization stream daily, and similarly, changes from the RTC
integration stream should be delivered to the RTC synchronization stream
daily.

This ensures that you can detect early when branch conflicts are
occurring between the RTC integration stream and the CC integration
branch (they will be detected by the nightly sync process). It also
ensures that changes will be sync'ed daily, so that you don't see
thousands of changed files being detected by the synchronizer (unless
you have thousands of files being changed in a single day).

Accumulating changes for long periods of time without synchronizing is
never a good idea ... just as having a development stream staying
isolated from its integration stream for a long period of time is never
a good idea.

Cheers,
Geoff

On 5/31/2010 10:22 PM, joseph wrote:

Hi Ralph,

Think the scenario that we have several dev streams and one
integration stream, one release stream. And release stream is the CC
synchronization stream. The changes will be accumulated to a certain
level then sync to the other side. Thousands files changed is
frequently in real case.

Beside, the statement "CC Connector is just suitable for
synchronizing files less than 20,000" cdlee mentioned is too
optimistic. The sync speed of CC connector goes down as file
increases. The project with less than 5,000 files may consider to use
it.

okay, then the only way to use CC connector is to sync it daily? otherwise, we need to endure the poor performance?

this time, maybe you should put the following into your product limitation,

"sync daily, otherwise, don't use it ...."

Regards,
Makson

No, "sync daily otherwise don't use it" would not be valid guidance.
Some old release branches are only used infrequently, and do not require
daily syncs.

On the other hand "don't wait until you have 10's of thousands of files
changed on an existing sync stream before doing a sync" would be valid
guidance.

Please see the earlier referenced tech note for accurate guidance:
http://jazz.net/library/techtip/68

Cheers,
Geoff

On 6/1/2010 6:38 AM, cdlee wrote:

okay, then the only way to use CC connector is to sync it daily?
otherwise, we need to endure the poor performance?

this time, maybe you should put the following into your product
limitation,

"sync daily, otherwise, don't use it ...."

Regards,
Makson

so "when to sync" is based on the number of files you needed to sync, not business requirement?

there are so many scenarios, you can not ask user to sync data just because the performance would be poor if there are lot of files.

maybe you should think "sync often" as a workaround for your current product, not a best practice.

Regards,
Makson

When developing a software tool, there are often are cases where
performance tradeoffs that must be made. In general, when a performance
choice must be made, we focus on optimizing the performance of "best
practice". In this case, "best practice" dictates that when a software
component is being developed in parallel in two software streams, that
those streams be synchronized as often as possible (to avoid divergence
and complex merge conflicts). We therefore have optimized the
performance of frequent synchronizations (i.e., the choice of what
behavior to optimize followed from the best practice, not the other way
around).

There certainly are cases when business requirements force you to not
follow standard best practices. Perhaps you could indicate why you feel
that applies to your particular case? I.e., why can't you follow the
advice from the earlier posting, that allows you to frequently
synchronize the synchronization streams (to avoid divergence and complex
merge conflicts), while flowing the synchronized changes back to the
integration streams only at appropriate (stable) intervals?

Cheers,
Geoff

On 6/1/2010 10:52 PM, cdlee wrote:

so "when to sync" is based on the number of files you needed
to sync, not business requirement?

there are so many scenarios, you can not ask user to sync data just
because the performance would be poor if there are lot of files.

maybe you should think "sync often" as a workaround for your
current product, not a best practice.

Regards,
Makson