Jazz Integration Architecture Overview

Draft version 0.24 - Last modified Sep. 22, 2008

The goal of the Jazz Integration Architecture (JIA) is to enable diverse tools to be used together providing an integrated experience to their users. This document provides an overview of JIA, and introduces the notion of a Jazz Team Server (JTS) that provides the Jazz Foundation Services (JFS) that allow groups of tools to work together. It also explains the relationship between JIA and Rational’s Open Services for Lifecycle Collaboration (OSLC) initiative.

Introduction

Motivation

IBM Rational’s customers use many software lifecycle tools in the course of developing their software and systems, only a portion of which are purchased from IBM. These tools (both IBM and non-IBM) have a variety of user interfaces and data stores – browser clients, rich clients, local file storage, relation databases, proprietary databases, etc. This diversity would not be problematic if the functionality from these varying technologies does not need to interact, and the data does not need to be related in any way. Everything works fine as long as each product is used in isolation to others. The difficulty is that this is almost never the case. A customer's development process often requires the usage of many different tools to accomplish their goals, and these processes make implicit connections between the data in the various tools.

Let’s consider a typical organization employing a range of tools from multiple vendors – or even the same vendor. This is often complemented by internally-developed custom tools. Of course, they want traceability between the resources across the development lifecycle like requirements, tasks, source code, and test cases. However, rather than uniformly connected resources, they often find integrations through specific bridges between each pair of tools – brittle connections based on unique tool-to-tool APIs. Moreover, their data is often buried inside the tools. When one tool needs to access another tool’s data, a bridge is required, implemented through a vendor-specified API often tied to a specific platform or language. And when a tool needs to record additional information, yet another bridge is required. This tightly coupled network of custom bridges can be vulnerable to everyday disturbances – changes like upgrades of the underlying OS or API revisions from the vendors. Additionally, tools tend to each have their own vocabularies, providing alternate names and descriptions for comparable concepts (or sometimes: different tools all using the same term with subtly different meaning). Even when tools can share data, they may be unable to share meaning, and a single logical asset can be scattered over multiple tools, requiring (even more) custom bridges, translation and synchronization.

The core problem above is that many lifecycle tools are built in silos. They provide user interfaces and data stores to implement particular functionality; they are not really designed to interact with other tools. The difficulty is that customers use these diverse tools together, and really need them to be designed in a way that allows them to work well together to solve their problem. The silos represent a painful artificial barrier.

The following might be a potentially ideal scenario for an organization. This organization still has a wide range of tools from multiple vendors. They also still have some internally developed tools customized for their specific processes. The difference is that these tools have all been designed to be used together – these diverse tools are coupled to form an ecosystem of interacting technologies.  For example, product management identifies requirements using a web based tool from one vendor. This data is linked to use cases and designs in a modeling tool from a different vendor. All these artifacts are associated with corresponding change tickets and governed by a process. Users routinely define relationships and traceability between data stored and managed by different tools. These relationships allow navigation between the user interfaces of the various tools. The data from any of these artifacts is available in well-defined formats and accessible to any other tool. This ideal scenario blurs the lines between individual tools. It sees software solutions more from the customer’s perspective because it is not only focused on the individual solutions but also on enabling an entire portfolio of interacting solutions.

Inspiration

Consider the World Wide Web. Each web site has web pages. Web pages are identified and accessed by their URLs, and can embed these URLs as hyperlinks to other web pages on the same or a different web site. The web browser presents a web page, and allows the user to navigate to other web pages by following the embedded hyperlinks. The desktop clients running the web browsers and the web servers that host the web sites are connected via the internet and converse via the standard HTTP protocol. The user gets to see the World Wide Web as a cohesive unity, knitted from the web pages from separate web sites. Web site authors can readily create web sites and web pages that integrate well with other web sites and web pages. Putting their documents up on a web site in one of the web's standard supported representations allows the content of these documents to be consumed by the widest range of users. And all this is achieved with relatively loose coupling between the web sites, allowing each web site the freedom to grow and evolve independently of other web sites. And despite having a relatively simple architecture that limits a client to sending a request to a server and getting back a response synchronously, people continue to find effective, performant ways to exploit it in various directions that were difficult to predict in advance.

The World Wide Web serves as the inspiration for our approach to solving the problem of integrating tools in such a way as to provide an integrated experience to users. It inspires in a figurative sense, in that we are looking for an architecture with many of the same characteristics as the Web. But it also inspires in a literal sense, in that our architecture adopts HTTP and some of the same technologies that the World Wide Web is built from.

JIA overview

The goal of the Jazz Integration Architecture (JIA) is to enable diverse tools to be used together providing an integrated experience to their users. JIA is not another monolithic platform but rather it is a set of inter-connected technologies and specifications. JIA consists of a reference architecture, API specifications, a set of common services and tool building blocks. It addresses the need for building new tools and JIA enables existing tools to be quickly integrated.

At the center of JIA is the Jazz Team Server (JTS). The JTS provides foundational services – Jazz Foundation Services (JFS) –to enable groups of tools to work together. These services include user and project administration, security, collaboration, query, and other generic cross-tool capabilities. When installed, tools are associated and work in conjunction with a particular JTS.

Powering much of JIA are standard RESTful APIs and standard resource definitions which enable participating tools to easily share data. JIA also includes reusable building blocks that speed the development of new project and the adoption of existing tools. Finally, it includes specifications that enable better integration of, and navigation between, the user interfaces of the various tools.

RESTful web services

In order to make its data and services available to the widest possible range of clients, a tool exposes its data and services through a resource-oriented web service [Richardson and Ruby, RESTful Web Services, O’Reilly, 2007]. The intent of the web service is to provide a stable long-term programmatic web-based API for directly accessing the facilities and data offered by the tool. These web services use an arrangement of URIs, HTTP methods, and standard representation languages such as XML and JSON, that work like the rest of the web. The protocol is stateless; client state is carried in the client; server state is explicitly reflected in the resources.

A REST API provides three key things: stable URLs for the tool data resources; documented representations for those data resources; and a protocol and operations for manipulating those data resources based on standard HTTP methods. Providing URLs for the tool data resources enables these resources to be linked from anywhere – a characteristic that distinguishes REST from SOAP/WS-*. In effect, the data resources become “hyper-data” just like hypertext enables fully connected, flexible content – text, images, etc. As with all APIs, REST APIs must be carefully designed with both stability and future evolution in mind. The long-term stability of URLs is important; without it, broken links becomes a serious issue. The long-term stability and compatible evolution of resource representations is what makes it possible to write clients that will work with data resources from a wide range of servers, possibly at different version levels.

Resource-oriented web services can easily provide a simple CRUD (Create, Read, Update, Delete) interface to resources using the POST, GET, UPDATE, and DELETE methods. POST can also be used to provide API operations with a more procedural flavor without compromising the benefits of using a purer resource-oriented style where that makes sense.

Jazz Team Server

Each Jazz Team Server (JTS) provides the foundational services which enable a group of tools to work together as a single logical server, and includes any number of Jazz Team Server Extensions that provide the tool-specific functionality. All of the foundation and tool-specific services are RESTful web services.

Jazz Team Server

All interactions between tools funnel through the REST API. To ensure that the tools are loosely coupled, they treat all URLs (except their own) as opaque strings, and make no assumption about where other tools are deployed. The same should hold for all clients, since we also want clients to be loosely coupled with the services.

JIA allows a single JTS to have multiple client-facing servers, each responsible for managing its own URLs and responding to requests coming from clients. Since the URLs identify the client-facing server on which the resource or service is hosted, clients’ requests may make direct contact with any of the client-facing servers. Internally, the tool coordinates its activities with its Jazz Team Server (again, by making behind-the-scenes calls to other services through their REST APIs). There need not be a single "front door" through which all requests from clients get funneled. This makes it easier to integrate tools with existing client-facing servers, and means that each tool can manage its URL space without having to coordinate with other tools. Client-facing servers may also include web UI servers dedicated to presenting data to users.

Jazz Team Server Extension

A Jazz Team Server Extension is tool-specific functionality which is affiliated with a particular JTS instance. For example the Rational Team Concert and the Rational Quality Manager products add functionality to JTS as server extensions. Each tool uses certain facilities provided via JFS in order to interact within the JTS, and exposes its own services and data through RESTful web services. The lifecycle resources – such as test cases, work items, or requirements – exposed in these RESTful services are also expected to align with the standards prescribed in Open Services for Lifecycle Collaboration (OSLC). These, when combined with the REST services of JFS, give a Jazz Team Server a uniform way of exposing all available functionality to clients. This allows clients to treat a Jazz Team Server as a unit - as something that offers a set of interconnected REST services - without having to be concerned with how those services are being provided.

A single JTS may consist of one or more servers that act together. The Jazz Foundation Services are typically hosted on a single server. JTS Extensions may be written as web applications (i.e., packaged as a WAR file) and deployed into a suitable server (possibly, but not necessarily, the one hosting JFS). Some JTS Extensions may deploy their own separate server. And some JTS Extensions may install tool-specific functionality simply by registering declarative configuration files and scripts via specially-provided foundation services. This means that tool developers have a spectrum of options for how tools can be packaged for deployment to a JTS, and ensure that customers will have flexibility when choosing server deployment topologies that suit their needs.

Jazz Team Server - logical vs. physical

An existing tool which gets adapted to be a JTS Extension may come with its own server and backend database. This implies that the data for a JTS may in fact be held in several databases. Supporting atomic transactions across multiple databases is known to be expensive.  Where necessary, tools will provide transaction facilities for their own data. JIA does not provide a transaction mechanism.

Jazz Foundation Services

The Jazz Foundation Services (JFS) are a concrete set of RESTful web services – REST APIs – for dealing with user and project administration, security, collaboration, query, and other generic cross-tool capabilities. Although many of the details are still to be worked through, the following sketch paints a slightly more detailed picture of kinds of services that the Jazz Foundation Services will provide.

Discovery Service

The Discovery Service URL locates a REST API for discovering the Jazz Team Server's various services and specific capabilities, such as a way to discover a web UI presentation for particular (kinds of) resources. The Discovery Service is an open-ended mechanism that can be used to keep track of and share important information about the Jazz Team Server. There are backstage mechanisms for the various services to register their important information with the Discovery Service. The Discovery Service is also used by all services to discover the whereabouts of other services provided by the other tools affiliated with this JTS. (Telelogic TDS and JRS's Discoverability Resources are early precursors of this kind of mechanism.)

Administration Services

Making tools feel integrated to their users requires integration both at the UI and behind the scenes, on the server. As with other distributed systems, each Jazz Team Server hosts a set of core Administration Services for dealing with users, projects, security, and licenses. For instance, the individual tools access these services behind the scenes to provide common authentication that can also be tied in to the organization's LDAP server. Another benefit is that users and projects can be centrally administered in a uniform way, rather than each tool having to provide these capabilities itself. (Antecedents of the various Administration Services are found in Jazz Platform 0.6 kernel and Telelogic TDS.)

Process Services

Of the various services, the principal area of Jazz innovation is in the realm of process guidance. By making the facilities of the Jazz Process Engine available as a service with a REST API, each of the individual servers can be process-enabled and participate fully in process guidance, including checking operation permissions and licenses, and determining process-specified preconditions and follow-up actions. This service also logs successful completion of all operations, providing a centralized audit trail for operations performed in accordance with the governing process. Users appreciate being able to configure a team's process centrally, and benefit from having each tool help reinforce the team's workflow on an ongoing basis. (The antecedents of the Process Services are found in the Jazz Platform 0.6 Process component.)

Storage Services

The Jazz Team Server provides Storage Services that tools can use to store their data, instead of having to create and manage their own database. (The JRS Storage Service is a precursor to this service.)

Query Services

Various clients need to search the data resources stored on a Jazz Team Server. This requires an approach that can handle the possibility that some tool-specific resources may live on a Jazz Team Server Extension deployed as a separate server. Jazz Team Server Extension servers are not necessarily homogeneous, and each might store its resources in its own database. The approach used is to extract each tool's data into searchable indexes, to consolidate those indexes across all tools deployed to a Jazz Team Server, and to provide centralize Query Services for searching across the consolidated index with queries expressed in a suitable query language. For structured information about resources, the extracted indexes are RDF-based and the standard query language is SPARQL. For full text searches, the mechanism is based on Apache Lucene. (Antecedents of both of extracted indexes supporting both structured queries and full text searches are found in JRS.)

Presentation Services

Well defined resource types and standard RESTful APIs provide the necessary backbone for JIA but it is not sufficient to enable the kind of multi-tool integration scenarios that are envisioned. The difficulty is that these resources are replete with links to other resources which may be managed by other tools. These linked resources may not be familiar to the client tool or this tool may not be able to provide a user interface for theses resources. For example, a work item may be linked to a requirement. It is unlikely that the work item tool would be familiar with the schema of requirement resources much less be able to provide complete UI functionality for it.

JFS Presentation Services

Presentation Services enable a client tool to request a user interface URL for any resource URL hosted by a JTS instance (including any of its extensions). When presented with a resource URL, this service returns a list of available user interface URLs which can be invoked for this resource. (Telelogic’s Application Registration was designed in part to enable this kind of functionality).

Data Warehousing Services

A corporate data warehouse may collect data from one or more Jazz Team Servers as well as from other data sources. The individual tools of a Jazz Team Server may take a snapshot of their data periodically and export the information to the corporate data warehouse via Data Warehousing Services. (The antecedents of these services are the Team Reports component of the Jazz Platform 0.6, Telelogic Altair and the Vega enterprise data reporting work.)

Collaboration Services

The Jazz Team Server provides other core services supporting collaboration, including services for sending email and SMS, maintaining subscriptions, etc.

Open Services for Lifecycle Collaboration

Open Services for Lifecycle Collaboration (OSLC) is an industry initiative, initially proposed by IBM in June 2008, aimed at simplifying collaboration across the software delivery lifecycle. The goal of the initiative is to enable teams to use disparate tools and share lifecycle resources in delivering software, whether the tools are from IBM, other vendors, open source projects, or in-house development. The OSLC initiative suggests an integration architecture and set of web protocols and services for lifecycle collaboration based on RESTful principles. (More information, including an initial set of descriptions for lifecycle resources such as requirements and test cases, as well as protocols and services for accessing these resources can be found at open-services.net.)

A tool that manages and stores lifecycle resources is expected to follow OSLC for its particular domain. For example, a tool such as the Rational Requirements Composer (RRC) product or the Telelogic DOORS product should expose requirement resources and services in accordance with OSLC. Doing so gives arbitrary OSLC-aware clients a way to access the requirements resources managed by that tool, without the client having to know anything about JIA, and independently of whether the tool is acting as a JTS Extension.

Similarly, tools that need to interact with lifecycle tools are expected to use the OSLC-specified protocols and services for that other tool's particular domain. For example, a tool like the Rational Quality Manager (RQM) product that provides a feature for reporting a bug when a test case fails should follow the OSLC-defined protocol for creating defect reports. This would allow the tool to create a defect against any OSLC tool that supports the OSLC standard for work items, with a link from the resulting defect report back to the failing test case. The server on which the defect is created could be a JTS, but it could just as well be some other kind of server entirely.

In summary, OSLC addresses what lifecycle services should look like to clients, but does not otherwise say anything about how they should be implemented. JIA goes beyond OSLC, to address building a JTS that implements OSLC-compliant services by loosely tying together multiple separate OSLC-compliant tools deployed as JTS Extensions.

UI composition

JIA will also include UI-related specifications that will enable tool-specific UI widgets from various tools to be assembled into larger composite applications. (This is demonstrated by systems like Lotus iWidget; Telelogic Team Webtop with widgets for Telelogic Change, DOORS and Tau; and by RTC’s Web Dashboards.)

Jazz frameworks

The term Jazz frameworks is meant to cover all language-specific libraries, frameworks, and other building blocks useful for adapting existing tools to support various degrees of JIA integration, or for writing new Jazz products. These would include language-specific libraries for accessing various Jazz Foundation Services and domain-specific services, and for building Eclipse, web, and command line clients.

These Jazz frameworks are considered to be productivity aids, rather than essentials; all degrees of JIA integration are achievable in any programming language capable of processing HTTP and XML. The client- and language-neutral REST APIs provide the foundation for developing language-specific client libraries for any language. Since the REST APIs are documented, the information needed to write a client library for any programming language is freely and publicly available. Jazz frameworks for some popular languages will be developed and maintained at jazz.net; but we anticipate others to be developed elsewhere.

How a Tool Implements JIA

The previous section described the key elements of the Jazz Integration Architecture. This section is aimed at tool writers; it looks at how these elements are used in practice when building a new tool, or adapting an existing tool, to implement JIA.

The ideal situation is for a tool to support the full complement of integration options offered by JIA. However, full integration needn’t be the starting point, especially for existing tools. Pragmatically, it’s likely to be more important that an existing tool starts off providing the kinds of integrations that will afford customers the biggest bang for the buck, and gradually improves the degree of integration over time. Regardless of whether adapting an existing tool, or writing a new one, there are a number of different ways for your tool to integrate with other tools using JIA. The degree to which your tool integrates is ultimately a function of what the tool does and what kinds of integration provide value to customers. Some key ways for your tool to play in JIA are as follows.

Provide a REST API

Your tool should expose its data and services through a REST API. If your tool manages lifecycle resources in a particular domain, the resource representations and web service should be OSLC-compliant for the domain. Once your tool does this, other tools can begin linking to your tool’s data resources via their URLs. New presentations can be created, relying only on details divulged in the REST API. And others can begin writing add-on tools that analyze your tool’s data resources, again, without needing to know more than what is revealed through the REST API.

Note that a tool can support a REST API without also having to participate in a JTS. Indeed, this is a reasonable configuration for tools that must ship as standalone products and work without being affiliated with any JTS.

A separate JIA document [Steve Abrams, RESTful Guidance] contains guidelines for REST API design.

Support UI navigation

Your tool should support Presentation Services so that your client – either rich client or web client – may by be invoked by consumers of your resources.

Interact with other lifecycle tools via domain-specific OSLC web services

If your tool needs to interact with another lifecycle tool, it should support doing so though via the OSLC web services for that other tool’s domain. For example, it your tool wants to create a work item, it should do so using the OSLC web services for work items. In this way your tool will be able to work against any work item tool that follows the applicable OSLC standards.

Register as a JTS Extension

An installed instance of a tool can be affiliated with an installed instance of a Jazz Team Server by registering it as a Jazz Team Server Extension. This static relationship is set up by an administrator, usually when the tool is installed. When the tool has its own server, the set up step entails configuring the tool’s server with the identity of the Jazz Team Server, and adding the identity of the tool’s server to the Jazz Team Server’s list of affiliated extensions. Although having your tool’s server registered with a Jazz Team Server provides no direct value to users, it is a technical prerequisite for most of the other kinds for integrations.

In some cases, a tool may be written to work exclusively as a JTS Extension and rely on the Foundation Services that a JTS provides. This tool could be shipped to customers and installed as an extension of one of their existing Jazz Team Servers (or, for customers who don't have one, a JTS install could be bundled with the tool). In other cases, it may make sense for the tool to be able to work without the support of a Jazz Team Server, and also be able to capitalize on a JTS when available. This would allow the customer to use the tool standalone, or to affiliate it with a Jazz Team Server if they have one.

Use JTS authentication

Your tool’s server is on the hook to secure access to its data and services. This is true whether access is via its REST API or some other means (e.g., a special-purpose interface). However, it’s especially important in the case of the REST API, since there is nothing protecting the server from unreasonable requests. The JTS provides a general user mechanism (part of the JFS’s Administrative Services) and LDAP based authentication. Your tool’s server should limit access to its data and services to users that have been authenticated via the JTS.

Contribute entries to JTS’s Discovery Service

Each Jazz Team Server has a central registry of information that is exposed to clients (and JTS extensions) via the Discovery Service, allowing clients to determine key property settings and locate resources and services hosted on the JTS. A static set of tool-specific entries can be included to the Jazz Team Server’s central registry when your tool is configured as a JTS Extension, allowing your tool to manifest itself to clients through the Discovery Service.

Process-enable operations

A Jazz Team Server provides a notion of process area with an associated team process that governs all activities, artifacts, artifact relationships, and operations that are pursued within the context of that process area. Customers teach the JTS about their processes, allowing the JTS to help the teams in following them. A process-enabled tool is one that is designed to play well in a world where a process known to the system governs the work being done. Process enablement entails opening up a tool so that it can be guided by process rules. Your tool should associate its resources with process areas. Then, when your tool handles a request to perform an operation on a resource, it should consult the Jazz Process Engine (one of the Administrative Services) to discover any process guidance associated with that kind of operation in the process area associated with the resource.

Utilize Storage Services

Your tool should start using the JTS Storage Service to persist its resources instead of storing these resources in a separate database.

Product license management

Your tool should utilize the JTS’s Administrative Services for dealing with product licenses.

Make data globally queryable

Your tool should participate with the JTS’s backstage indexing mechanism so that appropriate index entries get extracted for its data resources. This allows clients to find the tool’s resources by running queries with the JTS’s Query Services.

Snapshot data to data warehouse

Your tool should be prepared to snapshot its data periodically and export the information to the corporate data warehouse via the JTS’s Data Warehousing Services.

Write UI widgets

Create user interface widgets to view and manipulate your important resources. These widgets can then be incorporated into composite applications.