Questions about storing binaries in SCM and as attachments in RTC
We want to do continuous build and store the compiled binaries / libraries as part of streams
Are there any disadvantages of storing compiled binaries are part of streams ?
Oracle sure can hold bigger data size but I would like to know if there are any disadvantages of storing them in streams.
Some time back I read that it is not recommend to store large file sizes as attachments. Both attachments and scm are part of DB so I am curious to know about this
Thank you!
Accepted answer
Binaries and object files and the like can be stored in Streams. For example the JKE Banking example stores the ECJ compiler in one component. Some customers also want to store other development tools and libraries there.
This is acceptable if it changes infrequently.
My 2 cent on your case: If you store each build with all binaries in RTC, you will only make the hardware supplier of your hard drives happy. You will fill your database with a lot of data that should not be in there and you will overload your system and make it slow.
I would suggest other solutions solutions like an network storage to store the binaries for each build.
See: Build Artifacts Publishing Using HTTP Servers in Rational Team Concert
and
for some ideas.
Comments
As per the description above, I would not store the binaries in the build results either, but rather refer to them as demonstrated.
Attachments on work items should be used for documentation relevant to the work item, screenshots and the like. You don't want to attach a full release code for a build there either.
1 vote
Thanks Ralph.
The reason I brought in "Attachments" is just to understand the difference between data types stored in DB for Work items and SCM.
Of course we don't intend to store them as attachments.
Also we use File system for storing build results due to the size of the build output
One main reason for storing the libraries in RTC SCM is to share them instead of source code and make developers build always. If they have prebuilt libraries then local builds are faster
I understand the reasoning an I wish there was a better answer I could give. I would not try to put them into SCM. I would look into Binary Repository Managers such as Artifactory and Nexus to support the build process.
I have no experience myself how to integrate them into the build process, but these names have come up in discussions.
1 vote
Note, external libraries and the like, where you don't have the source code, would likely be put into SCM, but they change very infrequently. I just don't believe that RTC SCM would handle the load of binaries checked in each time. For source, it can find the delta and store that compressed. There is no way to do that for binaries.
Another problem is, where to store them. You don't want them in the development stream, because you would not even realize a file is not build in certain error scenarios, because the old binary is still there.
1 vote
Thank you Ralph.
To understand better why is it not recommended to store the binaries in RTC SCM? they do behave like every other file types right ?
Binaries can not be merged and behave different in storage. See https://jazz.net/forum/questions/73442/rtc-compression-for-binary-files-vs-clearcase
1 vote
scm is a lousy place for binaries as noted.
You should use Nexus, or Artifactory.. these both have facilities to get specific versions or 'latest'.
If you use maven or gradle as the build mechansim, then they both leverage these library repositories.
for non-maven builds you can also use ant plugins or curl/wget scripts to push/pull artifacts.
both repos are also supported by the leading deployment tools Jenkins, Urbancode, Ansible, DeployXL and can trigger deployment orchestration, all SCM independent.
also, binaries really mess up the file loading to users/build machines, as they are significantly bigger than source files and have a tendency to cause buffering problems in the web app server running RTC. this is one of the reasons we implemented the squid caching proxy servers to prevent repeated large file loads from impacting RTC performance.
1 vote
Nexus also has te ability to create 'Staging' repositories, so you can build continuously, into either snapshot or lowest staging tier, and promote without recompile as the code moves thru the test cycle.
Maven Release forces a recompile to move from snapshot to release, which is always fraught with danger, IMHO
2 votes