What's repotools -onlineverify?

Self-answered by Glenn:

Here's some more detail on the types of problems repotools verify find versus the types of problems onlineverify finds

The repotools -verify command reports errors detected by the application. Each component can look at the artifacts that it understands.  The verify command looks to see if the tables make sense in the context of the application.

Verify Examples

Every Contributor (user), must have a details record (User Name, photo...) associated with it.

Project areas names can't have special characters. See http://www-01.ibm.com/support/docview.wss?uid=swg21993070

For each source project area that has a link to a target project area, the target project area must exist. The target project area must point back to the source project area.

Versions of records created in configuration aware project areas must have unique creation times.

In configuration aware project area, only one version in a given stream can be the current version.

A given project area's development line references must logically sound. The development line that is referenced must exist, and must be considered part of the project area.

In SCM, a changeSet is not allowed to appear more than once in a workspace's history.

Online Verify

In CLM several tables hold duplicated data. These can get out of sync. Here are the tables that are verified: ITEM_STATES, ITEM_TYPES, ITEM_CURRENTS, CONTENT_STORAGE, and query tables (e..g FRIENDS, DASHBOARD, MODEL - a workitem),

Background

Every time you save a workitem, project area, requirement, or other artifact on CLM, it's saved into the repository into the tables mentioned above.

Consider a workitem that is saved. All of the fields in the workitem are saved in the ITEM_STATES table. This includes both fields that you can query on and those that you can't.

Every time you save a workitem, a new entry is created in the ITEM_STATES table. CLM also keeps all the older versions  – the history of the workitem - in the ITEM_STATES table. To keep this history as small as possible, CLM stores larger parts of the workitem (e.g. an attachment) in CONTENT_STORAGE. So, if you have a large attachment associated with a workitem. CLM doesn't resave the attachment each time you save the workitem.

The query table MODEL holds the latest entry of the workitem. The MODEL table is queried when you run queries to search for workitems. In general, queries only look at the most recently saved workitem.

Examples

Most of the issues are related to data inconsistency. Think of onlineverify as a tool that finds data inconsistency in tables with duplicated data. The errors look like this. See also https://jazz.net/wiki/bin/view/Deployment/DeploymentTroubleshootingVerifyCommand

The error below indicates that the query table (e.g. MODEL in the previous example) doesn’t have the same workitem as the ITEM_STATES table.

The Item with ItemID "_3Ac5BCdeFgHIjkLm7nO6pQ" and StateId "_-vUbAMclEdeQfJlWgh3ijk" was found in the ITEM_STATES table, but the same Item had a different StateId, "_Y64-zMyxEwvQuJlWts3rqp", in the Item query table.

Here’s another type of inconsistency. The XML for an ITEM is corrupt, or it’s not the expected type

2017-04-08 23:49:32,344 The ITEM_STATE row with itemId = _xCDJABn-EeemLZ7uEhrfcA, keyId = _F_sAuxn_EeemLZ7uEhrfcA and itemType = com.ibm.team.workitem#WorkItem failed to be demarshalled.

Database transactions generally prevent these sort of errors. These problems are more likely to be a result of database recovery done incorrectly, or some sort of database administration error. That is, one table was recovered from a backup but another wasn't.

Not all errors that are reported by online verify indicate that the product won't work correctly. The errors have to be looked at one by one.