Replacing work flow from component 'non-standard' svn model
( Apologies in advance for the length )
This is related to topic 'Importing from svn when a 'non-standard' model is in use' which is here http://jazz.net/forums/viewtopic.php?t=13572. In that thread I described how we have organized our code in subversion and we established that our history cannot really be imported into RTC. In this tread I would like to set aside the issue of history import and focus on how we would go about a recreating our workflow in RTC 'from scratch'. To begin with I'll point out that the iteration trunk folders in our svn repository are special. Examples of these would be /component x/source/trunk/iteration/51/trunk, and /component x/binaries/trunk/iteration/51/trunk. The iteration trunk under the binaries folder represents a manifest of deliverable files. If a developer needs to make additions to or removals from the set of deliverable files then he needs to check out binaries trunk, make the change, and check it back in. By doing this nothing gets added or removed by accident. Also by keeping the binaries in subversion - we don't have to maintain a separate system for housing builds - svn's use of binary deltas keep the storage costs down. - build integrity is maintained. That is if component X builds against component Y then in component X there will be an svn:externals ref to the binaries of component Y The iteration trunk under the source folder contains only completed features and bug fixes. When a developer is going to make a change he copies the iteration trunk to a story folder and then makes his changes under the story folder. Once he is done lets the 'svn admins' know and they then - check out the iteration trunk - merge the story folder into their workspace - build the code ( building includes run of tests ) - check in the merge results into the iteration trunk If there are any problems the 'svn admin' gets in touch with the story author and they work it out. In this way we are are generally confident in the quality of iteration trunk. Another thing that makes the source iteration trunk special is that the pre and post commit hooks. These watch for changes to the source iteration trunks. The precommit hook : - makes sure that only members of the svn admin group can make changes. If the user making the change is not a member then the it aborts the operation and delivers an appropriate error message. - checks to see if there is a build definition for the component. If there is then makes sure that the target build machine has enough space to perform a build. If there is not enough space then the it aborts the operation and delivers an appropriate error message. The postcommit hook checks to see if there is a build definition for the component. If there is then : - it scans the area of the repository under component X to determine the next build number ( see diagram in other post ) - it checks out the component's binaries from the corresponding binaries iteration trunk - it exports the revision of the source that triggered the execution of the postcommit hook over the binaries - if the build succeeds checks in the new binaries and creates folders with the build number under the iteration folder of the source and binaries folders. ( see diagram in other post ) - sends a mail out with the build status I believe the basic mapping of the structure we have in svn will need to look like this : <pre> component X folder --> component X stream component X source folder --> component X source component component X binaries folder --> component X binaries component A story folder --> repository work space </pre> So my first question to the RTC team is, does this look like an appropriate mapping? Next, I am having a problem with setting up the build trigger. Notice that in our current system the revision of the code that will be built is determined at the point where the build is initiated. This is important because it means you know exactly what is to be built. I don't see how quite how to set this up in RTC. I was thinking that I could potentially sent up an event handler that responds to the creation of a snapshot. However I don't see any examples of how one would do this. Also when I select the 'save snapshot' event in the project definition editor I see that the 'add' button is disabled. In other words I cannot tell if it is possible to set up such an event handler. I would appreciate it if someone could tell me the event handler is a good way to go and if someone could point me to an example of how to set one up. Next, I don't know if RTC will deal well with storage of the build results. First of all, in my testing it seems that RTC does not store deltas of binaries like SVN does. ( It does seem to use compression ). Say this because I checking in a build of one of our components and then updating what I had checked in with the subsequent build of the component. What I saw on my server was that the disk usage increased by the same amount each time. Could someone on the RTC team please let me know for sure whether binary deltas are used? On a rleated note I am having a hard time determining if there is a way to periodically clean up intermediate ( i.e. unshipped ) builds. You see with subversion we can use dump, dumpfilter, and load to remove all the binary history and then through a script just put back into the repository a history that is just made up the shipped binaries. What I would like to know is given my model up above ( binaries stored in a separate component ) will I be able to perform the same clean up that I can with subversion? Thanks in advance for any feed back. |
8 answers
Dan,
I don't know if I can address all your questions but I will do the best the i can: W.r.t your repo layout, it sounds to me like you would want a component to contain your source files. The component in the stream would only contain "blessed" changes while the component in user's workspaces could contain what they are working on (what you call a story). W.r.t storing binary content, RTC SCM does make use of deltas. However, we do not currently provide a way to delete historical content from the repository. This may mean that your current binaries strategy is not practical with RTC. If you did go that route (i.e. put your binaries in the repo), you could do it in a separate component, as you suggested. W.r.t builds, what we tend to do is create a snaphot as the first step in the build process in order to know exactly what was included in the build. However, I do not believe we have build triggers. Instead, I believe we use scheduled builds with a filter to indicate whether a build is required (i.e. we don't build if nothing changed since the last build). (Disclaimer: I am not an expert on the RTC build framework). W.r.t deliver permissions, etc, RTC has a process framework that allows for a role based definition of who can perform the various operations on a stream. However, we do not have the concept of pre or post commit hooks that would allow for the status check on the build machine or the triggering of the build as you described. Hope this helps, |
|
Thanks for the reply.
Ok, so that was what I was trying to convey my little mapping table. Glad to hear that I am on the right track.
If we were to use a separate component for the binaries, would removal of the component from a stream free up space?
Don't think this is equivalent to what I need. What I mean is that is the schedule mechanism means you end up with a build that has some 'random' set of changes in it and a mechanism that makes sure that the set of changes are labeled once the build process begins. This is a bit different then knowing up front what is to be built. This is also way is I was looking into setting up a handler for the 'snap shot saved' event. I was thinking that we could change our process so that the people that are the 'svn admins' today we create a stream snapshot and that this would then be the trigger for the build. Anyhoo, I'll keep digging.
Hmm, this raises another point. Today our pre-commit hook makes sure that developers don't put in empty check in comments. How would one do this with RTC? Thanks again. |
|
W.r.t. freeing up space, the answer is no. The component will still be in the repo along with the artifacts. I've been told that the ability to delete content is a high priority so this may change in the next release.
W.r.t. empty comments, RTC has process advisors that can be configured to ensure that change set has a comment and/or a work item associated with it before it is delivered to a stream. |
Hi Dan,
Your summary above suggests that you essentially have a separate branch for each iteration, that each developer works in isolation on a particular story at a time within the current iteration, and that these changes aren't merged into iteration branch until complete. Then, at the end of the iteration you merge all changes back into the main branch (or maybe there is no main branch, just the last iteration). In Jazz, we generally use separate streams for separate branches (i.e. split stream development). So for your scenario, one approach would be to create a separate stream for each iteration (copying it from the previous iteration's stream). Then, as each developer works on a story, they can keep their changes local to their development workspace. To run test builds, they can use the personal build feature to have an existing build definition build using the contents of their personal development workspace instead of the build workspace (and without accepting changes from the stream, as is normally done). If multiple developers work together on a story, they could create a separate stream just for that story. That way they can share (i.e. deliver) change sets between themselves, but still maintain isolation from the rest of the team. Personal builds can be run against the story stream. It's also possible to deliver/accept changes directly to/from another developer workspace, or to accept changes from an associated work item. It's often simpler to just work that way if there are only a few people involved, using the work item to track discussions and notify about the availability of changes. (Note that a change set needs to be marked as complete before others can accept it.) Personal builds are always manually requested, though. If you want to have automatic builds for a given iteration (or story), then you would need to set up a separate build definition, with its own dedicated build workspace having the iteration stream (or story stream) as its flow target. For the source and binaries division, you could either use separate components, or separate folders within a single component. Since they co-evolve fairly tightly, I don't see a big benefit in splitting them. |
|
Thanks for the reply Nick.
Hi Dan, For the most part this is correct. However once the changes are in the iteration trunk that is it. There is no subsequent merge. i.e. There is no 'main branch'. Getting back to the topic of builds, let me first say that I am not interested in personal builds. Developers are expected to build code on their workstations. ( They certainly aren't checking in code without compiling it! ) So I really have no need for that. So what I am looking for is a mechanism by which I can say 'build that version of the code'. As I stated above we use pre and post commit hooks to make sure that 1.) Only svnadmins can merge 'blessed' changes into the iteration trunk 2.) At the point when the check in to the iteration trunk is checked in that revision of the code is built. And while I see that I can set up a scheduled build the problem is that it will end up just building what ever happens to be available at the time the build runs. Likewise I see that I can manually request a build, however then steps have to be taken to make sure that nothing changes the workspace between the time when the build is requested and the point where the build actually occurs. Neither of these are really what I need. And as I said, I thought that a way I could get what I need is to set up a handler for the 'snap shot saved' event. However I cannot find any examples of how I might actually set that up and I am not certain that is really the best way to go. So what I am really looking for is an example of how to set up a handler for the 'snap shot saved' event or an alternative mechanism for getting a build triggered in the way that I need it to be triggered. Switching topics, wrt using separate components for the binaries, you are correct in that the source and the binaries co-evolve. However the are a few reasons I believe I want them to be separate - 1.) Say a developer is working on component X. ( Here component is not an RTC component but a component of our product ) . Today he only checks out the code for component X as he doesn't need the binaries. ( He is going to build the code after all. ) So in order to make it easy for a developer to just check out the code it seemed best to keep it as a separate ( RTC ) component. 2.) Today we have a 'super component' that just has references the binaries of all of the other components. This allows us to deliver versioned sets of components. e.g. version 1.0 of the super component is made up Version 2.0 of component X and version 1.5 of component Y and so on Replicating this in RTC is easier if the binaries of each of our product components are stored in an RTC component. Then this 'super component' becomes just a stream that is made up of the appropriate base lines of the binary components. 3.) Today we have several library components that are referenced by several other components. If component X uses library Y then what you see today is that in component X's source tree there is an svn:externals reference to the binaries of library Y. Again, the best way to set this up seems to be have a binaries component. Thanks again for the reply. |
|
> 1.) Only svnadmins can merge 'blessed' changes into the iteration trunk
You can make this restriction by only granting permission to deliver (and replace components) to the svnadmin role in the process area associated with the stream. > 2.) At the point when the check in to the iteration trunk is checked in that revision of the code is built. > And while I see that I can set up a scheduled build the problem is that it will end up just building what ever happens to be available at the time the build runs. You are correct that there's a possible gap there, where other changes could get delivered and included in the build. The svnadmin could run a personal build against their integration workspace after delivering the changes. As long as they're careful not to make further changes until the build's load phase is completed, it will build the exact contents. To do better than this, we'd need to support building against a snapshot, and also be able to trigger the build based on the delivery (or snapshot creation) rather than making it time-based. There's no out-of-the-box support for either currently. It would be possible to script building against a snapshot by replacing the contents of a build workspace from the snapshot, using the SCM CLI. For noticing a new snapshot under the current time-based scheduling, one option would be for the build script to list the snapshots on the stream and see if the latest one differs from the last built one. This can again be determined using the SCM CLI. For more on the usage of the SCM CLI in Build scenarios, please see: https://jazz.net/wiki/bin/view/Main/BuildFAQ#ScmCli Regarding the source vs. binary components, the reasons you give are all good ones for making them separate. |
|
Thanks again Nick.
Btw, there's a question of mine that has been overlooked. I was wondering if a 'snapshot saved' event handler could be used to implement what I need. To you know where I can find documentation on, or examples of, such an event handler? |
|
This is documented at:
https://jazz.net/wiki/bin/view/Main/TeamProcessDeveloperGuide#Adding_New_Event_Handlers Which is part of the RTC SDK doc: https://jazz.net/wiki/bin/view/Main/RtcSdk20 https://jazz.net/wiki/bin/view/Main/RtcSdk30 |
Your answer
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.