Jazz Forum Welcome to the Jazz Community Forum Connect and collaborate with IBM Engineering experts and users

Doors next generation 7.0.3: Pull all attribute values for an artifact

Questions
Tags
Users
Badges
Joshua Baxter
(111●27●28●32) Oct 14 '24, 4:53 p.m.

I discovered this API for pulling data for an artifact: https://jazz.net/wiki/bin/view/Main/DNGReportableRestAPI . I tested it and it doesn't appear that it returns all the attribute values (values for all the columns) for the artifact. For example, the artifact has a "Contents" column which displays the text content for the artifact that represents the requirement's details. However, the value for this column was not  returned by the API as it was not in the <rrm:attributes> XML element returned by the API, only the value for the ID was in the <rrm:attributes> XML element. How do I retrieve all the attribute values for an artifact using the API?

0 votes

2 answers
3,783 views
0 votes

Accepted answer

Permanent link
Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER (64.6k●3●36●49) Oct 17 '24, 2:38 a.m.

There are the available API's: https://jazz.net/wiki/bin/view/Deployment/ELMProductAPILanding

For DNG there are several APIs. There are REST based Reportable REST and server APIs. There is an OSLC API. The documentation for the APIs can be found using the links. Some documentation might be lacking. OSLC is a standard and the standards documents can be found e.g. here: https://docs.oasis-open-projects.org/oslc-op/rm/v2.1/requirements-management-spec.html 

My blogs do not directly cover DNG, but OSLC and other APIs. The Authentication Wiki Entry Davyd mentions, is hopeless, I think. I spent some time creating https://rsjazz.wordpress.com/2021/10/15/elm-authentication/ which should be able to get you where you need be. https://rsjazz.wordpress.com/2021/09/29/using-the-ewm-rest-and-oslc-apis/ shows how the EWM REST APIs can be used. The principles apply to the DNG APIs as well and the article series should be able to help you getting started with DNG.

Joshua Baxter selected this answer as the correct answer

1 vote

Comments
Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Oct 18 '24, 3:58 a.m.
Please note, in the case of OSLC, unless an attribute is a simple type such as String or integer etc, that has a standard representation, the attribute value will be a URL that would allow you to get the information needed. This could be the case for other APIs too. So, I am not aware of an API that would provide all data in one go.

One other answer

Permanent link
Davyd Norris
(3.9k●2●20) Oct 15 '24, 1:12 a.m.
You'll need to give us more information here.

What was the actual call you used to retrieve the information? The Reportable REST API has a number of end points that give various subsets of detail so it's important to use the correct one. 

You can retrieve the schema for each end point by using the ?metadata=schema parameter at the end of the endpoint, and this will give you the layout of the returned results.

The end point resources is the most generic and will get you high level lists of artefacts. The text endpoint is the one you want if you know the individual artefact URI or ID - this will pull all the details of an artefact including all attributes and the Primary test, both rich text formatted and plain text.

1 vote

Comments
Joshua Baxter
Oct 15 '24, 8:12 p.m.
Sorry I forgot to specify the end point I was calling. I was calling the resources end point. I'll try calling the text endpoint, thanks. Also, for parameters such as resourceURI and moduleURI which are used in many of the API calls at https://jazz.net/wiki/bin/view/Main/DNGReportableRestAPI, what is the maximum amount of URIs that can be passed into those parameters? Asking because want to know what the limit is before have to make a separate API call.

Davyd Norris
Oct 15 '24, 8:44 p.m.
As far as I know there's no limit, but for the text endpoint you'll either use the resourceURI or the artefact ID.  ModuleURI only applies to artefacts that are module format, and for those you'll need to get them via the resource or modules endpoints. For very large data sets, the returned results will be paged, with a default page size of 100 artefacts. You can change this number as part of your call but there's a hard limit set on the server so anything larger will be ignored. If you examine the returned results it'll tell you if there are more and give you details on the URL to call to get the next page

Joshua Baxter
Oct 16 '24, 8:25 a.m.
Thanks for your responses. I have a few more questions. How do I programmatically get resourceURIs and moduleURIs given that I have the artifact ID? Right now I only know how to get these manually. How do I programmatically get the desired configuration URL for the oslc_config.context parameter given I know the global configuration stream name? Are there any examples of how to programmatically call API's? For example, values needed for header, any sort of values/cookies/tokens needed to provide to the API for authentication, etc? Right now I've just been calling these APIs manually by copying the URL into the browser.

Davyd Norris
Oct 16 '24, 7:48 p.m.
This is a huge discussion! First the simple ones. 1. You can just use the artefact ID in filters - much simpler. Have a look at the API end point examples in the wiki and you can see how they are used. For things like modules, they return a list of artefacts in them and some basic info from the artefact. Each entry also has an about field which contains the resource URI, so you can use that to create a  call to the text endpoint. 2. If you're not using GCM or Local CM, you can leave off the context for many of the calls. IF you are then that's a new discussion. 3. There are a number of blogs, entries in the wiki, workshops, examples, etc. around how to be a well behaved Jazz/ELM client. Ralph's blogs are great, as is the Deployment wiki: https://jazz.net/wiki/bin/view/Main/NativeClientAuthentication

Joshua Baxter
Oct 17 '24, 5:59 p.m.
When I call the text endpoint (For example: https://server//rm/publish/text?resourceURI=resourceURI1,resourceURI2&oslc_config.context=configContext) it returns raw, unstructured text instead of XML which is unfavorable for processing. However if I call the schema for the text endpoint, https://server/rm/publish/text?metadata=schema, the schema it returns is structured XML so I'm confused as to why the text endpoint is returning raw text

Davyd Norris
Oct 17 '24, 9:20 p.m.
Very weird - I've never seen raw text come back. Do you have any headers set? At very least I would set: Accept: text/json or application/xml Content-Type: application/json or application/xml That may be why you're getting raw text back EDIT: I've just tested the call for a resourceURI on my own server and, even with no headers set, I get XML returned. I think the problem may be with your code. Can you try something like POSTMAN?

1 vote

Joshua Baxter
Oct 21 '24, 2:53 p.m.
I got it to work now using Postman and I get back XML; I forgot to specify XML in the header. Thanks for all your responses. You mentioned that "the default page size is 100 and that can change this number as part of the call but there's a hard limit set on the server so anything larger will be ignored" How do I change the default page size when calling the API? I don't see anywhere on https://jazz.net/wiki/bin/view/Main/DNGReportableRestAPI where it is specified how to specify the page size in the API call. How do I determine how many remaining pages there are for results returned by an API, as well as how to grab the next change? Also didn't see this specified in the documentation. What is the hard limit set on the page size for Doors Next Generation 7.0.3 server?

Davyd Norris
Oct 21 '24, 10:21 p.m.
Best way to see this is to make a call like https://<server>/rm/publish/text/* and watch what happens in the return. 1. Add the size=<number> parameter to the call. Default is 100 2. The response data will contain a ds:datasource entry, which will contain an href attribute that specifies the currently fetched data page, its size and the page number. It also has a rel attribute that will contain next if there's another page and an rrm:totalCount attribute that shows the total fetched, which is one less than the size parameter. Grab the href and increment the page number, and then send that request to get the next page 3. Write your code assuming the page size is out of your control - check the rel and rrm:totalCount attributes and fetch the next page using the adjusted href attribute if required

1 vote

Joshua Baxter
Oct 22 '24, 5:02 p.m.
"If you're not using GCM or Local CM, you can leave off the context for many of the calls. IF you are then that's a new discussion." We are using GCM and Local CM. I'm not sure how to programmatically retrieve the desired configuration URL for the oslc_config.context parameter. For example, in our widget we would want to allow the user to choose their desired local stream in DNG and then based on that selection, grab the configuration URL so that it can be passed into the oslc_config.context parameter for DNG Reportable REST API calls. Are there APIs that can be used to accomplish this?

Davyd Norris
Oct 22 '24, 6:41 p.m.
Hang on a sec - are you trying to build a dashboard widget? As in a Widget that can go on the mini dash in the RM application, or on a project area dashboard? I thought you were building a stand alone application. Depending on what you're trying to do, there's a whole lot easier ways to do this if you're building widgets to be used in ELM! You don't even need to log in It may be appropriate to select a correwct answer here and start a new question, but state in detail what your actual goal is - there may be a whole lot simpler way to do what you need

Joshua Baxter
Oct 22 '24, 7:24 p.m.
I appreciate all your help. Building a dashboard widget that can go on the mini dashboard in the RM application or on a project area dashboard. The goal is to create a dashboard widget that will allow users to compare all the artifacts in a specific module in one stream with all the artifacts of that same module in another stream and then see the differences. That's why I asked the above questions regarding how to get all details for all artifacts that are in a specific module via API calls via the "resource" DNG Reportable API and "text" DNG Reportable API. I know that you can compare streams within DNG, but we want to implement our own "comparison widget" due to the desire for additional functionality that is not in DNG's stream comparison functionality.

Davyd Norris
Oct 23 '24, 6:51 p.m.
OK if you want to do that then you will need to do it via the Reportable REST or OSLC APIs like you're doing. The RM Client Extension API is designed to be used in the currently active UI window, so only has access to the current configuration. Out of interest, what do you feel is missing in the current baseline/stream compare?

Davyd Norris
Oct 23 '24, 6:53 p.m.
BTW, one of the IBM Business Partners out there has already built something that sounds like what you want: https://softacus.com/extension-articles/doors/compare-module-across-configurations

1 vote

Joshua Baxter
Oct 24 '24, 1:34 p.m.
We are aware of Softacus' widget. We are actually trying to emulate that. Unfortunately we can't use Softacus' widget due to security reasons, which is why we are trying to create our own.

Joshua Baxter
Oct 24 '24, 2:28 p.m.
Your mention of the client extension API made me realize that I can probably use https://jazz.net/wiki/bin/view/Main/RMExtensionsAPI703#GetContentsAttributes or https://jazz.net/wiki/bin/view/Main/RMExtensionsAPI703##GetContentsStructure for getting all artifact information within the selected module for the current configuration and then use the Reportable REST API to get the information for the configuration that will be comparing to. Do you know if using the client API to get all artifact information within the selected module in the current configuration would be faster than using the Reportable REST API ? 

Joshua Baxter
Oct 24 '24, 2:32 p.m.
Also, I'm assuming that there's no way for me to test using the client API on my local machine since that requires the library for the client extension that is passed into widgets via DNG's OpenSocial framework?

Davyd Norris
Oct 24 '24, 7:24 p.m.
Couple of things:  - the Softacus widget is just Javascript that runs locally in the browser so I'm sure you could negotiate with them to host it locally rather than on their server. That would be the fastest and cheapest option if it does what you need  - both approaches are going to be very similar in time as they both will need to fetch the same info into your browser. The one difference is that the RM Client Extension is going to offer you an event based trigger so will let you be more responsive to user clicks and will return information like URIs that you need a lot more simply  - by definition the RM Client Extension API *is* running on your local machine as it's running completely inside your browser. As it's an event driven API, the only way you can test it is by triggering them in the DNG UI in your browser

1 vote

Joshua Baxter
Oct 28 '24, 4:07 p.m.
Thanks for all your help. There was some prior analysis done beforehand that concluded that we can't use the Softacus widget, and that we have to create the widget ourselves.  In order to speed up the process of pulling all artifact data within a module using the Reportable Rest API, we would implement generating multiple threads to make simultaneous calls to get different pages of data. For example: thread 1 gets page 0, thread 2 gets page 1, etc. Do you know if there is some type of limit in DNG server regarding how many simultaneous API calls it will process that it has received from from one IP address before it gets potentially flagged as something like a DDOS attack, and then potentially getting rejected? We may either do the API calls on the backend (via threads), or we may do them via multiple simultaneous asynchronous calls in the browser; haven't determine which approach we are going to take yet.

Davyd Norris
Oct 28 '24, 6:37 p.m.
You are using Javascript in the browser, which is single threaded, so you have no choice there. You also have to fetch the next page with the URL supplied in the previous call because there's a transaction for the fetches kept on the server in order to keep track of where you're up to

Joshua Baxter
Nov 04 '24, 3:34 p.m.
After testing calling the reportable API for getting artifact data that is within a specified module, looks like the maximum page size for our server is set to 1000. Since I now know what the maximum page size is, was going to use multiple threads to request pages simultaneously in the backend logic since based on that I could technically construct API requests to get different pages. When a request returns 0 results then that would indicate that there is no data for that page. The caveat with this approach is that if the maximum page size ever changes to another value other than 1000, then it would be very error prone so I see why you mentioned just using the URL supplied in the previous call. The problem is that our modules have thousands of artifacts, so doing it that way sequentially takes a few minutes.  Do you know if the "maximum page size" for API call could potentially change in the future, or is it a safe assumption to assume that the maximum page size will remain the same? Is that a configuration setting that we can configure to never change?

Davyd Norris
Nov 04 '24, 7:09 p.m.
It will absolutely change and can be tuned on the server, so you cannot rely on it. Also if you're doing this in a browser with JavaScript, then it's going to be single threaded anyway. The best approach is to do what DNG does itself, and displays the first page of info, then lazy fetches as the user scrolls back and forth.

1 vote

Joshua Baxter
Nov 19 '24, 12:27 p.m.
Discovered that using the Reportable Rest API, specifically the text endpoint, for pulling all artifact data inside of a module via a call like https://server/rm/publish/text?moduleURI=module&oslc_config.context=config&size=1000&pos=0 takes a long time if there are a lot of artifacts in the module. For example, we have a module that has 1660 artifacts and the API call above for pulling the first 1000 takes about 1 min 40s, and the total time to get data for all 1660 artifacts is about 2 minutes and 20s.  Is there a faster/more efficient way to pull artifact data? Is there a different API for pulling this data that is better regarding performance? We just need to retrieve the attributes' values and primary text. I didn't see any mention on Reportable Rest API, but I'm assuming there's no way to include a parameter in the calls to specify if only want to retrieve specific attribute values?

showing 5 of 22 show 17 more comments

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Ask a question
Guidelines
FAQ
Search context
Follow this question

By Email: 

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers
Answers and Comments
Question details
× 11,109

Question asked: Oct 14 '24, 4:53 p.m.

Question was seen: 3,783 times

Last updated: Nov 19 '24, 12:27 p.m.

Confirmation Cancel Confirm