Jazz Forum Welcome to the Jazz Community Forum Connect and collaborate with IBM Engineering experts and users

Questions about storing binaries in SCM and as attachments in RTC

Questions
Tags
Users
Badges
Karthik Krishnan
(909●19●130●170) Jun 03 '15, 4:08 a.m.
We use RTC 5.0.2 with Oracle as the back end DB

We want to do continuous build and store the compiled binaries / libraries  as part of streams

Are there any disadvantages of storing compiled binaries are part of streams ?

Oracle sure can hold bigger data size but I would like to know if there are any disadvantages of storing them in streams.

Some time back I read that it is not recommend  to store large file sizes as attachments. Both attachments and scm are part of DB so I am curious to know about this

Thank you!



0 votes

1 answer
6,481 views
0 votes

Accepted answer

Permanent link
Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER (64.6k●3●36●49) Jun 03 '15, 4:37 a.m.
You should be very, very careful what you store in an SCM system.
Binaries and object files and the like can be stored in Streams. For example the JKE Banking example stores the ECJ compiler in one component. Some customers also want to store other development tools and libraries there.

This is acceptable if it changes infrequently.

My 2 cent on your case: If you store each build with all binaries in RTC, you will only make the hardware supplier of your hard drives happy. You will fill your database with a lot of data that should not be in there and you will overload your system and make it slow.

I would suggest other solutions solutions like an network storage to store the binaries for each build.
See: Build Artifacts Publishing Using HTTP Servers in Rational Team Concert
and
Automated Build Output Management Using the Plain Java Client Libraries

for some ideas.
Karthik Krishnan selected this answer as the correct answer

2 votes

Comments
Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Jun 03 '15, 4:39 a.m.
As per the description above, I would not store the binaries in the build results either, but rather refer to them as demonstrated. Attachments on work items should be used for documentation relevant to the work item, screenshots and the like. You don't want to attach a full release code for a build there either.

1 vote

Karthik Krishnan
Jun 03 '15, 5:23 a.m.
Thanks Ralph. The reason I brought in "Attachments" is just to understand the difference between data types stored in DB for Work items and SCM. Of course we don't intend to store them as attachments. Also we use File system for storing build results due to the size of the build output One main reason for storing the libraries in RTC SCM is to share them instead of source code and make developers build always. If they have prebuilt libraries then local builds are faster

Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Jun 03 '15, 5:50 a.m.
I understand the reasoning an I wish there was a better answer I could give. I would not try to put them into SCM. I would look into Binary Repository Managers such as Artifactory and Nexus to support the build process. I have no experience myself how to integrate them into the build process, but these names have come up in discussions.

1 vote

Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Jun 03 '15, 5:53 a.m.
Note, external libraries and the like, where you don't have the source code, would likely be put into SCM, but they change very infrequently. I just don't believe that RTC SCM would handle the load of binaries checked in each time. For source, it can find the delta and store that compressed. There is no way to do that for binaries. Another problem is, where to store them. You don't want them in the development stream, because you would not even realize a file is not build in certain error scenarios, because the old binary is still there.

1 vote

Karthik Krishnan
Jun 03 '15, 6:24 a.m.
Thank you Ralph. To understand better why is it not recommended to store the binaries in RTC SCM? they do behave like every other file types right ?

Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Jun 03 '15, 7:39 a.m.
Binaries can not be merged and behave different in storage. See https://jazz.net/forum/questions/73442/rtc-compression-for-binary-files-vs-clearcase

1 vote

sam detweiler
Jun 03 '15, 7:54 a.m.
scm is a lousy place for binaries as noted. You should use Nexus, or Artifactory.. these both have facilities to get specific versions or 'latest'. If you use maven or gradle as the build mechansim, then they both leverage these library repositories. for non-maven builds you can also use ant plugins or curl/wget scripts to push/pull artifacts. both repos are also supported by the leading deployment tools Jenkins, Urbancode, Ansible, DeployXL and can trigger deployment orchestration, all SCM independent. also, binaries really mess up the file loading to users/build machines, as they are significantly bigger than source files and have a tendency to cause buffering problems in the web app server running RTC. this is one of the reasons we implemented the squid caching proxy servers to prevent repeated large file loads from impacting RTC performance.

1 vote

sam detweiler
Jun 03 '15, 7:57 a.m.
Nexus also has te ability to create 'Staging' repositories, so you can build continuously, into either snapshot or lowest staging tier, and promote without recompile as the code moves thru the test cycle. Maven Release forces a recompile to move from snapshot to release, which is always fraught with danger, IMHO

2 votes

showing 5 of 8 show 3 more comments

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Ask a question
Guidelines
FAQ
Search context
Follow this question

By Email: 

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers
Answers and Comments
Question details
× 11,105
× 1,210

Question asked: Jun 03 '15, 4:08 a.m.

Question was seen: 6,481 times

Last updated: Jun 03 '15, 7:57 a.m.

Confirmation Cancel Confirm