Jazz Forum Welcome to the Jazz Community Forum Connect and collaborate with IBM Engineering experts and users

How to request a list of artifacts in a component using Python REST API?

Questions
Tags
Users
Badges
Allison Schwoboda
(13●2●3) Dec 07 '20, 12:57 p.m.

 I asked an earlier question about querying DNG to find the artifacts for a project, and realized I needed to include a specific OSLC request in my query. However I am still having issues getting any information other than the folder names/structure. I receive a 403: Forbidden error with every request I make, however I am fairly certain the authentication is working as I'm able to see these folders as well as some other protected information. I have also tried using a single session object to make the requests, with no change in results. Here is an example of a request I'm making to see the artifacts inside a component:


       self.headers = {'Content-type': 'application/x-www-form-urlencoded', 'OSLC-Core-Version': '2.0', 'Accept': 'application/rdf+xml'}
info = {'oslc.select':'*' , 'oslc.where':'dcterms:identifier=123456' , 'oslc.prefix':'dcterms=https://ServerName/dc/terms/'}
        encoded = urlencode(info, quote_via=quote_plus)
        baseURL = 'https://ServerName/rm/views?oslc.query=true&projectURL=https%3A%2F%2FServerName%2Frm%2Fprocess%2Fproject-areas%2F_1ipDAGUlEemksZVbt6XKiw&'
        resp4_URL = baseURL + encoded
resp4 = s.get(resp4_URL,cookies=jar,verify=False,headers=self.headers)

And here is the response I'm receiving:

<Response [403]>
<rdf:RDF
  <rdf:Description>
    <err:detailedMessage rdf:datatype="http://www.w3.org/2001/XMLSchema#string"
    >CRRRS1602W  The operation is forbidden</err:detailedMessage>
    <err:errorMessage rdf:datatype="http://www.w3.org/2001/XMLSchema#string"
    >Forbidden</err:errorMessage>
    <err:errorStatus rdf:datatype="http://www.w3.org/2001/XMLSchema#long"
    >403</err:errorStatus>
  </rdf:Description>
</rdf:RDF>


Any help is greatly appreciated!

0 votes

Comments
Ian Barnard
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Dec 08 '20, 6:37 a.m.
Just noticed your baseURL = 'https://ServerName/rm/views?oslc.query=true&amp;projectURL=https%3A%2F%2FServerName%2Frm%2Fprocess%2Fproject-areas%2F_1ipDAGUlEemksZVbt6XKiw&'
which isn't well-formed - the &amp; should just be &, looks like you've double-encoded an &

5 answers
3,461 views
0 votes

Accepted answer

Permanent link
Ian Barnard
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER (2.4k●7●14) edited
Dec 08 '20, 6:30 a.m.

Hi Allison


1. Using a single requests session is definitely the way to go as far as I'm concerned - once your session is authenticated you won't have to do anything with cookies, i.e. less code, less opportunity for mistakes.

2. For my test server (6.0.6.1 iFix013) on an opt-out project (i.e. configuration management not enabled) I get a query base like this:
https://jazz.ibm.com:9443/rm/views?oslc.query=true&projectURL=https%3A%2F%2Fjazz.ibm.com%3A9443%2Frm%2Fprocess%2Fproject-areas%2F__NDNJ0NcLEeqXpuBdEolY7w
NOTE this doesn't end in &, where yours seems to - you need to confirm the full URL you are using is well-formed

3. A full query URL looks like this
https://jazz.ibm.com:9443/rm/views
?oslc.pageSize=200
&oslc.paging=true
&oslc.prefix=dcterms=<http://purl.org/dc/terms/>
&oslc.query=true
&oslc.select=*
&oslc.where=dcterms:identifier=3949
&projectURL=https://jazz.ibm.com:9443/rm/process/project-areas/NDNJ0NcLEeqXpuBdEolY7w
(apologies if the projectURL appears slightly different from the previous URL, this forum does something helpful/irritating with underscore characters, there's one at the start of the final part of projectURL)

I get two results (one is the core artifact, one is a module artifact)

NOTE the identifier must _not be surrounded by quotes - I tried this and got no results (but no 403)
NOTE the format of the value for oslc.prefix dcterms=<http://purl.org/dc/terms/>
The oslc.paging and oslc.pageSize are optional, and the server doesn't have to comply but I put them in anyway because it gives the opportunity when testing to speed things up by cutting short the query by setting the page size to e.g. 10 and stopping after the first page ;-)

Rather than using string concatenation, my python code takes the base query URL and uses urllib.parse.urlparse to split out the existing parameters, adds part 4 (the existing parameters from the base URL) to a dictionary of the specific query parameters then uses urllib.parse.urlunparse to correctly combine them back, this does the encoding at the same time :-) But as long as the final URL is well-formed that's really all that's needed.

4. Headers

My successful query doesn't have a Content-Type header, has the OSLC-Core-Version: 2.0 and Accept: application/rdf+xml but also has a vvc.configuration: https://jazz.ibm.com:9443/rm/cm/stream/_NMu5RtcLEeqXpuBdEolY7w (this is for an opt-out project, it doesn't give a 403 and results are fine if I don't provide this, but definitely would be required to get correct results for a configuration-managed project because it determines the component+configuration results come from)

Comparing to where you are in your question, I'd check:
1. What does your full query URL look like - are the parameters valid (first one with ? subsequent with &, that's a very easy mistake to make), is the format of each parameter correct (in particular dcterms), is it correctly encoded - you should be able to use this URL with a browser client like Postman, with the correct headers, and get results.
2. Check does your user in fact have permission to access the project?

Allison Schwoboda selected this answer as the correct answer

0 votes

Comments
Allison Schwoboda
Dec 08 '20, 3:08 p.m.

Thank you so much, this is very helpful. Could you explain how to obtain the vvc.configuration header?

Allison Schwoboda
Dec 08 '20, 4:25 p.m.

I was able to grab the configuration header manually from the database, but is there a way to do it programmatically?

Ian Barnard
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Dec 09 '20, 4:08 a.m.

:-) You use the Creation Factory for http://open-services.net/ns/config#Component to get the components in your project, which gets you a URL to get all the configurations in the component on oslc_config:configurations as rdfs:member, then get the config to see its name. That's the 10,000m view, gory details with an example of the RDF at each stage are in a previous answer of mine here https://jazz.net/forum/questions/266334/dng-oslcfetch-components-from-project-area. If you're going to be selecting a configuration by name then you might want to be defensive and raise an exception if >=2 configs have the same name, rather than just using the first match you find. Good luck!

4 other answers

Permanent link
Jim Amsden
(293●4●7) Dec 07 '20, 2:48 p.m.

 'dcterms=https://ServerName/dc/terms/' should be

0 votes

Comments
Allison Schwoboda
Dec 07 '20, 2:59 p.m.

Hi Jim,


I just changed this as you recommended and am having the same issue. 

Ian Barnard
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Dec 08 '20, 6:00 a.m.

Actually it should be

Permanent link
Jim Amsden
(293●4●7) Dec 07 '20, 3:16 p.m.

 try changing  'oslc.where':'dcterms:identifier=123456'  to  'oslc.where':'dcterms:identifier="123456"' 

0 votes

Comments
Allison Schwoboda
Dec 07 '20, 3:48 p.m.

 Just tried this, and still getting a 403. Is it possible the error is a result of something other than the URL used? I don't know if the headers could be a problem or if it's possibly a permissions issue with the database?

Permanent link
Jim Amsden
(293●4●7) Dec 07 '20, 4:42 p.m.
Check baseURL. It should be the queryBase for the project area. Follow rootservices to rmServiceProviders to get the ServiceProviderCatalog, lookup the ServiceProvider for your project area, and get its Services XML document. The queryBase should be in there.

Are you including a cookie store with your connection so that the authentication tokens are included in subsequent requests? Here's some sample code that gives some hints...


        self.base_url = server_url
        self.userid = user
        self.password = password
        self.spc = None
        self.sp = None
        self.ownerMap = dict()
        
        # Disable SSL so that we can read self-assigned certificates
        self.http = httplib2.Http(".cache", disable_ssl_certificate_validation=True)
        self.http.follow_redirects = True
        self.headers = {'Content-type': 'application/rdf+xml'}
        protectedResource = "/whoami"
       
        # Create an authentication challenge by attempting to access a protected resource
        resp, content = self.http.request( self.base_url + protectedResource, 'GET', headers=self.headers)
        if (resp.status == 404):
            # the JRS server does not support whoami
            protectedResource = ''
            resp, content = self.http.request( self.base_url + protectedResource, 'GET', headers=self.headers)

            
        if ('x-com-ibm-team-repository-web-auth-msg' in resp and resp['x-com-ibm-team-repository-web-auth-msg'] == 'authrequired'):
            # JEE Forms authentication
            resp, content = self.http.request( self.base_url + protectedResource, 'GET', headers=self.headers)
            self.headers['Cookie'] = resp['set-cookie'] 
            self.headers['Content-type'] = 'application/x-www-form-urlencoded'
          
            # now we can start the authentication via j_security_check page
            resp, content = self.http.request(self.base_url+'/j_security_check' , 'POST', headers=self.headers, 
                        body=urllib.parse.urlencode({'j_username': user, 'j_password': password}))
            # Successful login will set the LtpaToken2 required for subsequent authenticated access
            self.headers['Cookie'] = resp['set-cookie']
        else:
            # Try sending the credentials for OpenIDConnect
            self.http.add_credentials(user, password)
                         
        # check authentication was successful, if not throw exception
        resp, content = self.http.request( self.base_url + protectedResource, 'GET', headers=self.headers)
        assert (resp.status == 200), f"Login failed with status: {resp.status}"

0 votes

Comments
Allison Schwoboda
Dec 07 '20, 5:01 p.m.

Yes the baseURL I'm using is taken from the services.xml document. And I'm following this process for the authentication, but the cookies are kept in the requests session. Previously I was passing them as a parameter each time, but it was recommended that I use a single session instead. However I'm fairly certain this is working as I'm able to successfully access the folders which are a protected resource. 

Permanent link
Jim Amsden
(293●4●7) Dec 10 '20, 1:40 p.m.

dcterms:identifier is of type xsd:string in the OSLC specification, and the OSLC query specification says strings have to be quoted in order to distinguish them from booleans, and numbers. So quoting the dcterms:identifier="330" should work and does for me. It also works without the quotes, but that is not following the OSLC standard.


0 votes

Comments
Ian Barnard
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Dec 11 '20, 7:09 a.m.

Hmm, I can only get the double-quoted version to work if I explicitly specify a typed literal dcterms:identifier="3949"^^xsd:string

Ian Barnard
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Dec 11 '20, 1:13 p.m.

Ah seems like the 7.0.1 you're using doesn't need the ^^xsd:string to figure out that "3949" is a string ;-) My 6.0.6.1 does. Or for both, dcterms:identifier=3949 (i.e. without quotes) works just fine.

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Ask a question
Guidelines
FAQ
Search context
Follow this question

By Email: 

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers
Answers and Comments
Question details
× 10,936
× 478

Question asked: Dec 07 '20, 12:57 p.m.

Question was seen: 3,461 times

Last updated: Dec 11 '20, 1:13 p.m.

Confirmation Cancel Confirm