Database Growth - Strategies for minimizing the growth of repository databases

Authors: RosaNaranjo, AlaEddineBannour, TimFeeney, PaulEllis, DanielMoul, IngoUnruh
Build basis: None

In planning or maintaining your IBM Engineering deployment, you may ask yourself these questions:

• "How do I ensure that my database size will not grow in an unmanaged way?"
• "Are there any strategies I can put in place to help with this in the beginning?"
• "Are there objects I should avoid storing in my repository?".

There are some recommendations that will be captured in this page as well as some links to content that will help you understand.
We can break down those strategies into two approaches: approaches that you can apply to your current environment, and approaches that you should additionally consider when starting a new project or planning to deploy a new environment.

Reduction Approaches

The following approaches are applied by each application if you want to reduce the data in your existing environment:

In DOORS Classic

• Purging objects/modules
• Deleting baselines
• Deleting objects

DOORS Next

• Archiving Project Areas
• Archiving components
• Archiving Configurations
• Discarding of empty change sets. You can automate this approach with a script.
• Data Redaction: This approach is not intended for bulk deletion usage, and is only applicable for the redaction of sensitive data. This is unlikely to reduce the database size.

The "safest" way to reduce database size, prior to the 7.x release, was to archive projects and use the repotools -deleteJFSResources command. See the Deleting data permanently from a DOORS Next Generation project technote if this is required as part of an upgrade from a release prior to 7.x.

Engineering Workflow Management/Rational Team Concert

In Engineering Workflow Management, formerly Rational Team Concert, there are several items that can contribute greatly to database growth if not kept in check. These items are build results, workitem attachments, and binary content in SCM as versioned content. There are several articles and presentations that have been written on this topic.

• Deleting SCM components and all their histories
• Moving SCM components to a new server
• Deleting the builds results
• Storing the build outputs in an external repository
• Archving project areas

Additionally, there are several articles and presentations that have been written on this topic:

• Moving An Source Code Management Component To A New Server
• Help! My RTC Database is getting big!
• Follow up to Help! My RTC Database is getting big!
• Deleting data in Engineering Workflow Management
• Reducing the size of the Rational Team Concert repository database
• Deleting content in RTC

Engineering Test Management/Rational Quality Manager

For QM, there are some usages that can drive repository db growth. For example, automation generating Execution Results and attachments on the test artifacts such as HTML, images, etc. attached to either scripts or results. QM does have some ability to permanently delete content. See Deletion and restoration of test artifacts.
Other useful guides for Engineering Test Management are:

• Archiving project areas
• Archiving components
• Archiving and Restoring Configurations
• Deleting test cases results in bulk by creating a script using the API for deleting test cases results
• Deleting test cases execution results by creating a script using the API for deleting test cases execution results
• Deleting artifact attachments

Systems Rhapsody Design:

Removing diagram images from the server

In Lifecycle Query Engine (LQE):

Moving to LQE relational store will need significantly less hardware and resources ( in RAM /Storage)

Data Warehouse

Data warehouse database size is huge and growing fast

Lifecycle Index (LDX):

• Moving to LDX relational store
• Decommissioning LDX and using only LQE Rs in v.7.1

All of the approaches that include archiving Data will reduce a the Data that LQE is storing in its Index .

Data Management

In addition to the above approaches, there are other approaches that you can implement when setting up a new project to ensure data efficiency, and that only the useful data is stored in the system. These approaches are again broken down per application:

DOORS Classic

• Automated deleting of baseline using a DXL script

DOORS Next:

• Archiving project areas and team areas
• Making a baseline in the past
• Administrative setting to block uploading attachments above a defined size

Engineering Test Management:

• Automated deleting test cases results in bulk by creating a script using the API for deleting test cases results
• Automated deleting test cases execution results by creating a script using the API for deleting test cases execution results
• Blocking uploading attachments above the defined size

Engineering Workflow Management/Rational Team Concert

• Automatically deleting the builds results automatically
• Storing the build outputs in external repository
• Storing work item attachments in an external storage
• Blocking uploading attachments above a defined size
• Publishing/unpublishing configurations to LQE
• Deleting SCM components using a repository tools command

In Systems Rhapsody Design

• Publishing /unpublishing configurations to LQE

Jazz.net forum resources

The forum contains many good suggestions from ELM Administrators who are maintaining ELM deployments.

Question: I want to store binaries in EWM. Does EWM-SCM support delta compression?

Answer: Yes. EWM-SCM supports delta compression of files, including binary files. In fact, one of the properties you can change in advanced properties is whether to enable or disable delta compression. If delta compression results in larger gains than other compression types it will be used to compress a given binary.

Question: Can I prevent binaries from being checked-in to EWM?

Answer: Yes, you can implement file limits in EWM.

Question: Does EWM store all versioned elements in the database?

Answer: Yes, EWM-SCM stores all versioned elements in the database.

Question :What can I do to monitor db growth and what does EWM do on its own?

Answer:

• EWM will run item cleanups daily (sometimes frequently throughout the day) to help clean stale data. You can investigate increasing this task, or the expiration time of these items (located in Advanced Properties)
• You can also run the onlineverify to ensure the consistency of the data and that there is no corruption which may be preventing the item clean up from running.
• Ensure all your builds are being pruned and any results cleaned up

Question How can you scrub ChangeEvents from the repository database using IBM Engineering Workflow Management?

Answer: Engineering Workflow Management repository database ChangeEvents scrubbing

Additional posts of interest: Data warehouse db sizing

Are there any ways for me to reduce the size of the EWM db

RTC compression for binary files vs clearcase

RTC large history complexity and archiving questions

What is the Latest Metrics by Namespace report?

Rather surprising db2 growth

Expected database growth when versioning binary data

Versioning binary artifacts

Related topics: Deployment Planning and Design, Why is my database consuming so many resources?

Additional contributors: StefVanDijk, BenjaminSilverman, KrzysztofKazmierczyk