loading large components

While we do not have a hard limit restricting the size of a share
operation in Jazz Source Control, the larger the share you do, the more
rows get written in the database (potentially 7N for the N
files/folders), which adds contention to the system, a very long running
transaction, and potentially cause other problems like:
- content gets cleaned up before it is claimed in the checkin
- transaction log or buffer pools may unexpectedly become exceeded

That being said, the better hardware/disk that you use, the higher that
limit is. We are in the process of figuring out the means of
determining that threshold for the deployment. Once we have that data,
we can restructure our client applications to chunk large shares into
chunks that perform best for the system.

Thanks
JohnC
SCM Server

Ton van Velzen wrote:

I noticed today that a 10MB/2.000 files upload (sharing project) between
a 1.0.1.1 client and a server over an MPLS cloud was about 60x faster
(less than 1 minute) than a 150MB/20.000 files upload between the same
sites.

Something doesn't seem to scale linearly in RTC.

Can anyone reproduce this?
Any comments?

Thanks, Ton van Velzen

While we do not have a hard limit restricting the size of a share
operation in Jazz Source Control, the larger the share you do, the more
rows get written in the database (potentially 7N for the N
files/folders), which adds contention to the system, a very long running
transaction, and potentially cause other problems like:
- content gets cleaned up before it is claimed in the checkin
- transaction log or buffer pools may unexpectedly become exceeded

That being said, the better hardware/disk that you use, the higher that
limit is. We are in the process of figuring out the means of
determining that threshold for the deployment. Once we have that data,
we can restructure our client applications to chunk large shares into
chunks that perform best for the system.

Thanks
JohnC
SCM Server

Ton van Velzen wrote:

I noticed today that a 10MB/2.000 files upload (sharing project) between
a 1.0.1.1 client and a server over an MPLS cloud was about 60x faster
(less than 1 minute) than a 150MB/20.000 files upload between the same
sites.

Something doesn't seem to scale linearly in RTC.

Can anyone reproduce this?
Any comments?

Thanks, Ton van Velzen

So - it sounds like there might be some tuning on the database - buffer pools, etc that we could think about to cover this kind of situation.

I suppose the good news is that sharing a project (that initial upload with that many files) is a relatively rare event in the lifecycle of a real project - it just happens to be the one you see first :-)

anthony

Thanks for the reply. It would be good to be able to follow this work. Under
what workitem is this work being tracked? When is this client restructuring
expected to become available? Will it be part of the 2.0 release of RTC?

Thanks, Ton.

johnc wrote:

While we do not have a hard limit restricting the size of a share
operation in Jazz Source Control, the larger the share you do, the more
rows get written in the database (potentially 7N for the N
files/folders), which adds contention to the system, a very long running
transaction, and potentially cause other problems like:
- content gets cleaned up before it is claimed in the checkin
- transaction log or buffer pools may unexpectedly become exceeded

That being said, the better hardware/disk that you use, the higher that
limit is. We are in the process of figuring out the means of
determining that threshold for the deployment. Once we have that data,
we can restructure our client applications to chunk large shares into
chunks that perform best for the system.

Thanks
JohnC
SCM Server

Ton van Velzen wrote:
I noticed today that a 10MB/2.000 files upload (sharing project)
between a 1.0.1.1 client and a server over an MPLS cloud was about 60x
faster (less than 1 minute) than a 150MB/20.000 files upload between
the same sites.

Something doesn't seem to scale linearly in RTC.

Can anyone reproduce this?
Any comments?

Thanks, Ton van Velzen

Agreed that the upload of large data sets is the first thing you see. I don't
mean to be pessimistic, but downloading large data sets not only takes a
noticeable amount of time when you do it frequently (for multiplatform tests,
e.g.), but large data sets invite large/complex change sets too. I suspect that
updating rows is not less expensive than inserting rows. So, largeness will not
go away once you have loaded the big chunks up. Once you got it, largeness is
there to stay.

The notion of large components is a reality for many customers. They use RTC in
a way that RTC ideally is able to be tuned to. Large chunk usage patterns may be
quite different from the way other customers will use it. performance tuning is
a very good idea for large or atypical installations.

kesterto wrote:

johncwrote:
While we do not have a hard limit restricting the size of a share
operation in Jazz Source Control, the larger the share you do, the
more
rows get written in the database (potentially 7N for the N
files/folders), which adds contention to the system, a very long
running
transaction, and potentially cause other problems like:
- content gets cleaned up before it is claimed in the checkin
- transaction log or buffer pools may unexpectedly become
exceeded
That being said, the better hardware/disk that you use, the higher
that
limit is. We are in the process of figuring out the means of
determining that threshold for the deployment. Once we have that
data,
we can restructure our client applications to chunk large shares
into
chunks that perform best for the system.

Thanks
JohnC
SCM Server

Ton van Velzen wrote:
I noticed today that a 10MB/2.000 files upload (sharing project)
between
a 1.0.1.1 client and a server over an MPLS cloud was about 60x
faster
(less than 1 minute) than a 150MB/20.000 files upload between the
same
sites.

Something doesn't seem to scale linearly in RTC.

Can anyone reproduce this?
Any comments?

Thanks, Ton van Velzen

So - it sounds like there might be some tuning on the database -
buffer pools, etc that we could think about to cover this kind of
situation.

I suppose the good news is that sharing a project (that initial upload
with that many files) is a relatively rare event in the lifecycle of a
real project - it just happens to be the one you see first :-)

anthony