Jazz Forum Welcome to the Jazz Community Forum Connect and collaborate with IBM Engineering experts and users

Publish a snapshot back to a RTC build result from a Jenkins build job

Questions
Tags
Users
Badges
Rob Leach
(35●4●8) Aug 23 '17, 11:07 a.m.

 I am trying to publish a specific snapshot to a RTC build result. The build setup we use is as follows. We use Jenkins 2.x with Pipeline build scripts (pipeline script from scm). And the way we interact with builds is through build definitions setup in RTC (so developers submit build requests in RTC). The RTC build definition is pretty simple and just connects to a corresponding Jenkins build job. And as already mentioned the Jenkins build job uses a Pipeline build script from SCM.


The Pipeline build script uses a Jenkins global shared library using the following Pipeline syntax (on the first line of the Pipeline build script).

Library('dcbldlib') _

Then further down in the Pipeline build script there is a scm fetch (we use the General SCM checkout Jenkins Pipeline step). The fetch code looks like this:

checkout([$class: jazz.class, avoidUsingToolkit: jazz.toolkit, buildTool: jazz.tool, buildType: [buildStream: 'stream name', clearLoadDirectory: false, customizedSnapshotName: props.snapshotName, loadDirectory: props.loadDir, overrideDefaultSnapshotName: true, value: 'buildStream'], credentialsId: 'jenkins-creds-id', overrideGlobal: jazz.override, serverURI: jazz.uri, timeout: jazz.timeout])
Some builds have multiple scm fetches (one up to three is typical--these are in addition to the global shared library which happens at the very beginning of the build job). I can set (or override) a snapshot on each respective fetch (scm checkout) as you can see in the code example, and this works as expected. If I checkout a particular stream, I can confirm my snapshot is listed in the snapshot list for that stream.

However the problem is I cannot publish the correct snapshot back to the build result. From reading what IBM documentation I've found, I see that this property (or environment variable) is used for this task "team_scm_snapshotUUID". But when I try to publish the snapshot back to the build result I'm only getting the snapshot from the global shared library and nothing else.

How do I publish a specific snapshot back to a RTC build result (buildResultUUID)?
Or how can I get the snapshotUUID from a specific checkout to use to publish my snapshot back to a RTC build result?

This property, team_scm_snapshotUUID, only seems to contain the snapshot from the very first scm checkout (which in this case is the global shared library).

This is the task being used to perform the publish snapshot: com.ibm.team.build.ant.task.SnapshotPublisherTask

1

1 vote

Comments
Rob Leach
Aug 23 '17, 3:24 p.m.

 Exploring further I re-read the checkout step documentation and found this statement "This step returns a Map". Referring to the checkout Pipeline step itself.



However it appears that the RTC scm does not return a map, it appears to return nothing at all.

Rob Leach
Sep 06 '17, 1:56 p.m.

 Here is the RFE:

Rob Leach
Oct 25 '17, 3:18 p.m.

 I wanted to follow up on this in case the information is useful to others. We've recently started exploring using Git for our SCM repository (GitLab, GitHub, Bitbucket, etc.). As a preliminary proof of concept we've stood up a GitLab instance and started migrating some source to repositories therein. One of the first things we migrated were the Jenkins Pipeline build scripts, the Jenkins Global Shared Library, and other build support scripts (Gradle, Ant, etc.). When we did this and plugged in all the requisite Pipeline pieces, we noticed that the build property team_scm_snapshotUUID now has the correct snapshot UUID, it no longer has the snapshot UUID from the Jenkins Global Shared Library fetch (as I've described in other comments). So clearly it seems that the issue revolves around the Jenkins Global Shared Library fetch which in our case occurred at the very beginning of the build process (first thing).

Rob Leach
Dec 06 '17, 1:15 p.m.

 https://jazz.net/jazz/web/projects/Rational%20Team%20Concert%20(SAFe)#action=com.ibm.team.workitem.viewWorkItem&id=446243



Lakshmi Narasimhan T V
JAZZ DEVELOPER Dec 07 '17, 11:29 a.m.

Hi Rob,
Thanks for updating this post with the relevant work items.

With workflow-cps 2.40 and above, the env object is populated more than once. So the last checkout should have caused the team_scm_snapshotUUID to point to the snapshot of the stream. I have mentioned this in the wiki as well.

3 answers
4,853 views
1 vote

3 answers

Permanent link
Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER (63.7k●3●36●48) Aug 28 '17, 8:14 a.m.

I would suggest to contact support and maybe open an enhancement request. It might be possible to achieve what you want using the API, however it would be nice if it was supported in the Jenkins Build.

0 votes

Comments
Rob Leach
Aug 29 '17, 11:09 p.m.

 Ouch. Not the answer I was hoping for. But regardless I appreciate your thoughts on this topic. I'll follow your advice and have our Rational Team contact support and open an enhancement request.


I'll still hold out hope that someone might be able to offer a work around solution. Even as Ralph outlined, using the API.

Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Aug 30 '17, 2:05 a.m.

Rob,

an entrypoint into the API here is com.ibm.team.build.ant.task.tests.SnapshotPublisherTaskTests

com.ibm.team.build.ant.task.tests.SnapshotPublisherTaskTests.testSnapshotContributionAlreadyExists()

shows that part of the API.

Also search for Build Automation is https://rsjazz.wordpress.com/interesting-links/ and follw the links for other API examples.

If you are new to the API you want to follow https://rsjazz.wordpress.com/2015/09/30/learning-to-fly-getting-started-with-the-rtc-java-apis/

Rob Leach
Aug 30 '17, 4:06 p.m.

 Thanks.

Permanent link
Geoffrey Clemm
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER (30.1k●3●30●35) Aug 30 '17, 2:21 a.m.

Since you should be able to publish arbitrary build results back to RTC, can't you just publish the snapshot URL back to RTC as one of your build results?  (A file containing that snapshot URL string, for example).   It won't show up nicely in the GUI in the RTC build result, but the information would be there for you to use.

0 votes

Comments
Rob Leach
Aug 30 '17, 4:12 p.m.

 That is certainly an option. Again not ideal, but it would provide the information in the build result. As you mention, you can publish anything back to the build result, so a snapshot URL could be constructed and published back to the build result.


I'm sure I'll hear complaints about the snapshot not showing up as expected (as a valid build result snapshot), but as long as the information is available for the individual developers to consume. It would be something they locate in the build results links tab.

Thanks for the tip.

Rob Leach
Aug 30 '17, 9:42 p.m.

 Okay so I've established I can use this task to publish a link back to the build result: com.ibm.team.build.ant.task.ArtifactLinkPublisherTask

And I've established that this is the format of the URL for the snapshot: https://jazzcm.host:9445/ccm/resource/itemOid/com.ibm.team.scm.BaselineSet/_EN63uI3oEeejGvLjZrr0mA

But the last part of the URL is not known (this part: _EN63uI3oEeejGvLjZrr0mA), and I don't know how to get that. I've already established that the checkout for the RTC scm type does not return anything (contrary to the Jenkins checkout documentation).

It always seems to fall back to needing the snapshot UUID which is what I think this is, _EN63uI3oEeejGvLjZrr0mA.

Sigh.

Geoffrey Clemm
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Aug 31 '17, 4:04 a.m.

 I see a couple of alternatives:

- Use the RTC scm command directly to perform the snapshot create, rather than using the Jenkins checkout_scm command.
- Ralph's suggestion to file an RTC enhancement request against the Jenkins Plugin category, asking for the URL or UUID of the snapshot created by the Jenkins checkout_scm command to be returned in the map returned by the checkout_scm command.   Note that the documentation for the checkout_scm does not specify what should appear in this map, so you cannot count on anything specific being there (so the current behavior of returning nothing appears to comply with the spec for the command).

Rob Leach
Aug 31 '17, 10:16 a.m.

 Okay using the RTC SCM CLI is another option. This option would probably require more in terms of setting up the plumbing etc., but seems like it would offer more control (at least clearer expected outcomes).


I contacted our Rational Team support and explained the details of the issue. They decided to open a PMR (so that's done). I'll ask them why they chose that route....

Rob Leach
Aug 31 '17, 10:16 a.m.

 I know on the Jenkins side, the build job does "know" all the details about the RTC build information. When I review completed build jobs within the Jenkins console, all the snapshots are listed with the snapshot name as the label, and the URL of the snapshot as the link (and the correct snapshot UUID is included in that URL).


Again it always falls back to getting or accessing that snapshot UUID so I can consume it. The RTC Jenkins plugin is doing a lot of the right things, I just don't know how to get this one piece of information beyond what the documentation has to offer. Going the route of implementing my own library using the Java API seems excessive for such a simple use case (and somewhere under-the-covers that information is captured).

But you've both already stated an enhancement request probably seems the best course of action.

Thank you, both, again.

Permanent link
Rafael Rezende
(43●18●23) edited
Sep 04 '17, 2:47 a.m.

For some reason I cannot comment directly under your question or any of the answers. So, please don't take this as an answer :-)

I'm afraid I will fall into the same issue very soon, so I would like to know more about your configuration. I'm missing just few details on how you are checking out the Global Shared Libs and the Jenkins pipeline script.

This is how I would compose my environment...

Principles:
A - The tools around the build environment are "static". I don't want to get the latest state, I want a proven groovy code that handle/build my target project. Therefore, these tools are loaded from Snapshots.
B - The target project should be in its latest state. Therefore, Stream, RW or Build Definition (which in turn loads a RW).

Based on that,
1 - the Global Shared Libs are part of A, being loaded from a Snapshot.
2 - the Jenkins pipeline script is also part of A, loaded from a(nother) Snapshot.
3 - the actual project (B) is loaded using the Build Definition.

I would expect the Build Results for this configuration to contain a link to the corresponding checkout. That is, the Jenkins pipeline script (2) uses the Global Shared Libs (1) and triggers the checkout (3) using the Build Definition.
The last checkout is the one that creates the Build Result, so I would find very strange and counter-intuitive to have any Snapshot other than the one from (3).

Do you have a similar configuration?
If yes, which Snapshot do you see in your Build Result? 1, 2 or 3?
Would you have a Work Item number to share? I would happily vote on that.

Update: I'm able to convert the answer into Comment after posting it. Strange UX imho.
Update2: I can't edit a Comment after converting it from Answer. It has the "edit" button, but it pops a plain HTML page instead. Looks like bug.
Update3: I had to convert it back to Answer to be able to update the message. But now I'm unable to convert it to Comment again. Looks like another bug... The message will have to stay as "answer".

0 votes

Comments
Rob Leach
Sep 10 '17, 12:01 p.m.

 Not an answer...

Well I like the way you think Rafael, specifically your idea about having your global shared library and Jenkins pipeline script pulled from a snapshot. I think that's smart and a good idea. Unfortunately we are not at a point to implement that level of static control on either of those.



So we have a development center group (project area) in our Rational Team Concert (SCM) instance (separate from our product project areas). We have setup a stream and component for the global shared library and the Jenkins pipeline build scripts respectively in our development center project area (so two streams with their own unique component in each). Today we have configured our Jenkins master to pull each of those directly from their respective stream. But one day we'll likely stabilize and using a snapshot will make sense then.

So what you described is pretty much the way I see it too. But I wouldn't agree with this comment, "The last checkout is the one that creates the Build Result". I think that's an assumption that's not accurate in my experience. The build result is created on the RTC side of the wire, if I hadn't already explained, we start our build by submitting a build request from RTC (via a build definition). So I think when RTC communicates to Jenkins that it wants to run "X" build job, the build result already exists and all the relevant information is handed off to the RTC Jenkins plugin (I'm kinda speculating here). Based on the Jenkins build job configuration (using a SCM Jenkins pipeline build script) the first thing that is retrieved is the Jenkins pipeline build script (in our case from a stream), and that is loaded and executed. Within the Jenkins pipeline build script if you're using the global shared library, then that gets pulled down next--perhaps it depends on where that line is in the Jenkins pipeline build script, I'm not sure (we use this syntax on the first line: Library('dcbldlib') _)

Then the source code for the product being compiled is retrieved (further down in the Jenkins pipeline build script) with a regular pipeline checkout using the RTC SCM.

So the numbered bullets you listed above should really be in the following order: 2 first, then 1 and lastly 3.

It is my speculation that under-the-hood the RTC Jenkins plugin manages all the appropriate snapshots just fine, and distributes them to the requisite locations the RTC Jenkins plugin team has programmed it to (that's all good). But the issue is communicating specific details back to the original RTC build result. And in my case the snapshot of the product source code (I don't care about the snapshot for the global shared library nor the snapshot for the Jenkins pipeline build script). From reading the IBM documentation this property is supposed to contain the snapshot UUID, team_scm_snapshotUUID. It does contain a snapshot UUID, but it is the snapshot UUID for the global shared library checkout (fetch). So I have not found a way to get the snapshot UUID for other checkout operations. I should be able to be selective about which snapshot UUID I care about. I know the RTC Jenkins plugin knows about all the snapshot UUIDs, because it uses them to update the build job result in Jenkins. The developers are the ones that want to consume that information, and it seems like a fair request, and they only use the Eclipse client for their day to day work; most, if not all, the developers don't even know we are using Jenkins on the backend to do the heavy lifting. So publishing that information back to the build result in RTC is a consumption issue rather than a truly technical issue.

Someone at IBM on the RTC/Jenkins team knows what I'm referring to and would very easily have an answer on how you can selectively use "this" snapshot UUID, and not "that" snapshot UUID (or at least not the one that is being populated in the property team_scm_snapshotUUID). It's just a matter of the right person stumbling on this thread.

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Ask a question
Guidelines
FAQ
Search context
Follow this question

By Email: 

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers
Answers and Comments
Question details
× 10,927
× 151

Question asked: Aug 23 '17, 11:07 a.m.

Question was seen: 4,853 times

Last updated: Dec 07 '17, 11:29 a.m.

Confirmation Cancel Confirm