RegisterLog In to Jazz.net dW

Sample Server Source

Getting Started with the Server Sample

This document helps you install and get started with the sample server implementing the Open Services for Lifecycle Collaboration (home). This sample server is provided in source form, developed in Python and distributed with an open source (BSD) license. The server allows developers to be able to interact with an implementation of the services described by the Open Services for Lifecycle Collaboration initiative. The server implements these services, with the following details and/or caveats.

Service Comments
Discovery Resources The server implements a root Atom Service Document at /jazz/atomsvc.xml, OpenSearch documents for the search and query services as well as an Atom Service Document for published collections.
Simple Resource Storage The server implements the Simple Resource Storage service as described.
Collection Storage The server implements the Collection Storage service as described.
Resource History The server implements the Resource History service as described.
Resource Properties The server implements the Property resources as described with the exception that property selection is not supported..
Resource Archiving The server implements the Archiving specification as described.
Resource Change-Events The server implements the Change Events service as described with the exception that on POST of an entry to a collection change events are created for the new entry but a change event is also created for the update to the feed itself.
Full-Text Search The server implements the Full-Text Search service as described. Note that the updating of full-text indexes is an asynchronous operation and so there may be a time lag between the successful storage of a resource and it's appearing in search results. The supported query language is relatively simple in comparison with Lucene.
Structured Query The server implements the Query service as described using SPARQL as the query language on POST. In actual fact the server supports both the service described by the Open Services for Lifecycle Collaboration but also implements the W3C SPARQL protocol. Note that the updating of indexes is an asynchronous operation and so there may be a time lag between the successful storage of a resource and it's appearing in query results.
Resource Indexing The server implements the indexing-rules collection as described, however only certain elements in an indexing rule are applied during indexing. Currently the server only takes the "element" and "objectType" attributes of an index into account, not the "object" or "property" attributes and no secondary resources.
Users Feed The server implements the Users service as described, note that adding a user to the server is accomplished using the Django admin web UI.
Authentication, Authorization The server implements Authentication against the Django users database and uses HTTP Basic authentication. The server implements Authorization using Django permissions (see later).

It is important to note that the server sample is not intended as a poduction environment, while every attempt has been made to implement the Open Services for Lifecycle Collaboration faithfully, the sample does very little in terms of error recovery and no guarantees are made that it will not lose data. Specifically it takes a rather simple approach to managing connections to the full-text DB and RDFlib, opening and closing them for every request which can cause locking issues.

Installing the Server Sample

The sever is provided as a zip file, Drake-0.6.0.zip, and can be run directly from the unzipped source without any additional setup or configuration. The following pre-requisite packages need to be available before running the server (note that each of these packages has it's own separate license, these are included in the download zip file). Detailed installation instructions are at the end of this document.

  1. Python 2.5.x - obviously this is required to run the code. Downloads from the Python web site include packaged Windows installers and similar for other platforms. For Mac OS X 10.5 (Leopard) the pre-installed version of Python is useable.
  2. Django 0.96 - provides the framework on which the server is developed. This package is Python only and is setup in the same way as the Drake sample server above with a setup.py script.
  3. RDFlib 2.4.0 - provides indexing and SPARQL query services for the server.

The next step is to run the server's database creation tool, the first step is to change directory to the "drake" directory and find the script manage.py. Run this script with the command parameter "syncdb", as shown in the listing below; note that you'll have to create an admin user, if you want to run the unit tests provided with the package you should name this admin account "simon" and set the password to "simon" also.

$python manage.py syncdb
[15/May/2008 07:09:37] INFO version: 2.4.0
Creating table django_admin_log
Creating table auth_message
Creating table auth_group
Creating table auth_user
Creating table auth_permission
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table resource_atomfeed
Creating table resource_numbergenerator
Creating table resource_storedresource
Creating table indexingrules_xmlindexentryspecification
Creating table indexingrules_regexindexspecification
Creating table indexingrules_xmlindexspecification
Creating table history_resourcehistory

You just installed Django's auth system, which means you don't 
have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (Leave blank to use 'simon'): 
E-mail address: simon@ibm.com
Password: 
Password (again): 
Superuser created successfully.
Installing index for admin.LogEntry model
Installing index for auth.Message model
Installing index for auth.Permission model
Installing index for resource.AtomFeed model
Installing index for resource.StoredResource model
Installing index for indexingrules.XmlIndexEntrySpecification model
Installing index for indexingrules.RegexIndexSpecification model
Installing index for indexingrules.XmlIndexSpecification model
Installing index for history.ResourceHistory model
Loading 'initial_data' fixtures...
No fixtures found.

With the database created you are ready to run the sample server itself - f you want to find out more about running the server, for example using the Apache web server or running over SSL check out the Django documentation, for now we'll use the built-in test server. Note that by default all the server's data is stored in your operating system defined temporary directory (c:\temp, /tmp, ...) which on a Unix platform for example means that all data will be deleted if the machine restarts. To change the location of these data files you can find the configuration in a Python script settings.py in the same directory as the manage.py script. Similarly the server is configured by default to use SQLite as the database backend which can be changed in the same settings file.

$python manage.py runserver 8082
[15/May/2008 07:39:53] INFO version: 2.4.0
[15/May/2008 07:39:54] INFO version: 2.4.0
Validating models...
0 errors found.

Django version 0.96, using settings 'drake.settings'
Development server is running at http://127.0.0.1:8082/
Quit the server with CONTROL-C.

Your server is now ready, which you can test with your web browser, try some of these links:

If these work, you're ready to go, the next section demonstrates some ways to interact with the running server. Also, if you wish to turn debugging on, edit the file __init__.py in the same directory as the main script and change the logging configuration call.

# To enable trace set level to logging.DEBUG-1
logging.basicConfig(level=logging.INFO,
                    format='[%(asctime)s] %(levelname)s %(message)s',
                    datefmt='%d/%b/%Y %H:%M:%S',
                    stream=sys.stderr)

You can replace the standard value of "INFO" to either "DEBUG" for more messages or "DEBUG-1" to enable all the tracing information - be warned that's a lot of log.

The Server Sample in Action

One way to understand the Open Services protocols is to review the unit tests that are included in the server package as these exercise all of the services provided by the server. Also there are client-side samples provided by the Open Services for Lifecycle Collaboration initiative. Finally some of the behavior of the server can be invoked simply by using your browser. The following screen shots demonstrate some of the functionality of the sample server using just a Firefox session.

First we'll use the Django admin web UI to create a resource, as it's difficult to issue an HTTP PUT command from the browser. Enter the URL shown in the image below, substituting your server name and port if you are not using the default configuration. You'll see the logon form and enter the details of an admin user (we are using the admin user we created in the syncdb task above).

Admin Web UI Logon

Once logged into the Django admin web UI you'll see the following main page. The admin site allows an administrative user the ability to view and edit not just Django details such as users and groups but also application (in this case the Drake server) exposed details such as the resources stored in the server and various index specifications. For now let's simply add a new stored resource by either clicking on the Add button next to the Stored Resource line or clicking on the Stored Resource name and then the Add button on the following page.

Admin Web UI Main Page

The following form allows you to either enter a new resource or edit an existing resource stored in the server. We'll add a new resource and see what happens. Note that it is important that the URL id specified on this form starts with a forward slash "/" or the server will fail to include it in key services such as search and query. The body and content type may be anything you like as long as they are meaningful. At the bottom of this form you will see an Advanced options line, there are additional details that can be set for a resource, for instance to archive or un-archive a resource or set an expiration date/time -- do not use these until you understand the implications.

Admin Web UI Add Resource

When you press the Save button you will be returned to the list of stored resources. If you have already reviewed the list of stored resources before adding the glossary above you'll have seen the list empty and so may be surprised to see that not one but two resources were stored when you hit save. The second resource (the one with the horrible URL that messes up the UI) is a change event, denoting the creation of the new resource. This is a system resource but it is stored in the same table with all other resources.

Admin Web UI Resource Saved

If you wish you can see this change event from the Drake system change-event feed. Simply use the /jazz/services/changeevents URL path in the browser and you should see the following (obviously the rendering of feeds is browser/version dependent). Clicking on the link will show you the actual change event that should tell you the URL of the resource created and the user that created it. Note that the actions above used URLs rooted in /admin/ whereas the pages below are rooted in /jazz.

Drake Change-Events Feed

But more than just the creation of the change-event happened when you hit save, in fact a number of important operations where performed by the server:

  1. The change event is created on every save (create or update) or delete of a resource.
  2. The addition of a history record is performed on every save and delete of a resource.
  3. The resource is passed to the full-text search indexer to allow users to search for it.
  4. The resource is passed to the query indexer to allow users to run queries for it.

So let's see if the full-text search indexer has done it's job, to do this we will actually run a search. To perform a search in the browser use the URL /jazz/services/search?q=Project, note the query parameter "q" after which comes the term(s) you wish to search for. Specifying more than one term implies an AND operation and will only match resources where all terms match, you can override this with an explicit OR operator, as in "?q=Project+OR+Program" using the "+" character to correctly URL-escape spaces.

Drake Search Results

Clicking on the link should, as expected take you to the resource itself (the complete URL for which is /jazz/resources/example.xml).

Drake Resource

But what of the resource revision history - how does that work? Well simply adding the query string "?revisions" to the end of the resource URL from the last operation should result in a feed containing an element for each revision of the resource.

Drake Resource Revision

In a similar way you can inspect the properties extracted from the resource by the query indexer by adding the query string "?properties" to the end of the resource URL and you'll retrieve the RDF description for the resource as shown below. The properties shown are all system properties but all are available for query.

Drake Resource Properties

And, like the full-text search above this is possible in the browser too. Specifically the Drake server supports four query modes:

  1. A URL-encoded list of predicate=object terms, as specified by the Open Services for Lifecycle Collaboration service description.
  2. A URL-encoded SPARQL query, as specified by the W3C SPARQL protocol recommendation.
  3. A POSTed SPARQL query, as specified by the Open Services for Lifecycle Collaboration service description.
  4. A POSTed URL-encoded SPARQL query, as specified by the W3C SPARQL protocol recommendation.

Using the first of these we first enter the query URL /jazz/services/query and add our terms, in this case we wish to query for all resources whose content-type is "application/xml" so the complete term becomes "http://purl.org/dc/terms/format=application/xml", click here. Note there are two important issues when running queries of this form:

  1. Special characters such as "#" need to be escaped (in this case as %23), this character is common in term URLs.
  2. Predicates whose type is not string must have a type prefix, so to search for all resources created by simon you would need to use the URL http://localhost:8082/jazz/services/query?uri:http://purl.org/dc/terms/creator=/jazz/users/simon as the stored value is the URL to the user entry.
Drake Query Results

To include elements from your XML format in the query properties you'll need to create an index specification which you can either POST to the indexing-rules system feed or, for testing, you can use the admin web UI to create them using the form-based editors.

The server includes a set of test cases you can run (in the test directory). Also look for the client-side samples for examples on how to create and manipulate resources in the server. Another useful tool for debugging is the cURL command-line tool which is usually installed on most modern Unix (and Mac OS X) systems. cURL allows you to send requests from a command-line client, requests of the form:

1 curl http://localhost:8082/jazz/atomsvc.xml
2 curl -o discov.xml http://localhost:8082/jazz/atomsvc.xml
3 curl -u simon:simon http://localhost:8082/jazz/users
4 curl -u simon:simon -I http://localhost:8082/jazz/users
5 curl -u simon:simon -H "Content-Type: application/msword" -H "If-None-Match: *" 
       -T sample.doc http://localhost:9080/jazz/resources/sample.doc
6 curl -u simon:simon -H "Content-Type: application/atom+xml" -H "If-None-Match: *" 
       -T empty-feed.xml http://localhost:8082/jazz/resources/myfeed
7 curl -u simon:simon -H "Content-Type: application/xml" 
       --request POST --data-binary @glossary-example.xml 
       http://localhost:8082/jazz/resources/myfeed
  1. Performs an unauthenticated GET.
  2. Performs an unauthenticated GET and stores the results in a file.
  3. Performs an authenticated GET, otherwise this request would fail.
  4. Performs an authenticated HEAD (-I).
  5. Perfoms an authenticated PUT (-T) creating a new resource from the local file sample.doc.
  6. Performs an authenticated PUT (-T) creating a new feed using the empty feed XML content.
  7. Performs an authenticated POST (--request POST) using the binary data denoted by @filename, to the feed we created previously.

The Server Sample Internals

This section is not a detailed description of the implementation of the server sample but is intended to give enough information for those with an interest in the internals to find their way around. The server itself is a Django web application, even though the primary services it provides are not user-interface oriented. The server does benefit from the Django admin user interface however, as mentioned above. The server has a set of internal applications (in Django terms) which roughly correspond to the services described in the Open Services for Lifecycle Collaboration documentation. The server makes use of Django URL dispatching to loosely couple these services to the URLs clients use to access them; we also make extensive use of Django templates to generate Atom feeds/entries, search and query results and more. The full-text search is provided by a library that leverages new capabilities for text indexing in SQLite. To provide the structured query service (and associated indexing) the server uses the RDFLib Python package with data stored in SQLite again.

The server also uses the Django users database to implement authentication and authorization, all operations requiring authentication will perform the authentication check against the Django user database using the credentials supplied over HTTP Basic authentication. In terms of authorization the server uses the permissions capability in Django to model the roles defined in the Open Services for Lifecycle Collaboration documentation of Reader, Writer and Admin. Specifically the following permissions must be set for a user to meet the role requirements.

Role Django Permissions
Reader Can get Stored Resource
Writer Can get Stored Resource
Can add Stored Resource
Can Change Stored Resource
Can Delete Stored Resource
Admin Can get Stored Resource
Can add Stored Resource
Can Change Stored Resource
Can Delete Stored Resource
User MUST ALSO have the Django "is staff" flag set

Rather than having to add these permissions individually to each and every user it is more convenient to create groups in Django and assign the correct permissions to the groups (Drake Readers, Drake Writers for example) and then assign users to these groups.

Detailed Installation Instructions

  • For Mac OS X Users:
    • Mac OS 10.5 comes with Python 2.5.1 pre-installed so you don't need to install it yourself.
    • For Mac OS prior to 10.5 you'll need to download a version of Python 2.5.1 (or above) from here.
  • For Linux Users:
    • These instructions have been tested on Ubuntu 7.04, Intel 32bit. Ubuntu comes with Python installed, and Ubuntu 7.04 comes with 2.5.1 so no further install required.
  • For Windows Users:
    • Ensure you have a copy of Python version 2.5.1 (or above) installed, if not you can download it from here.
      • Once installed ensure you add the python (default c:\Python25) directory to the path (for example in a Command Prompt set path=c:\python25;%path%).
      • You'll also need to install a C compiler, the easiest way is using the MinGW compiler and tool suite.
  • Download the Django 0.96.2 source package from here, cd into the Django-0.96.2 folder and install using the command
    python setup.py install.
  • Unzip the Drake-0.6.0.zip file and cd into the Drake-0.6.0 folder.
  • To install RDFLib use the command
    python ez_setup.py prereqs/rdflib-2.4.0-py2.5-{platform}.egg
  • To install PySqlite2 use the command
    python ez_setup.py prereqs/pysqlite-2.4.1-py2.5-{platform}.egg
  • Now follow the sample server setup instructions above.

Note that Linux and Mac OS X users may need to preface the commands above with sudo depending on the the permissions of the user issuing the command.

Issues and Corrections

To report issues and corrections please use the Open Services for Lifecycle Collaboration mailing list open-services@jazz.net, or contact the author Simon Johnston, skjohn@us.ibm.com. Also check out the Open Services for Lifecycle Collaboration web site for new information and code updates.