DatabaseGrowth < Deployment

Engineering Lifecycle Management Wiki - Deployment
Community information and contribution guidelines
TWiki > Deployment Web > DeploymentPlanningAndDesign > DatabaseGrowth
Revision 23 - 2025-10-17 - 15:55:23 - PaulEllis
<div id="header-title" style="padding: 10px 15px; border-width:1px; border-style:solid; border-color:#FFD28C; background-image: url(<nop>https://jazz.net/wiki/pub/Deployment/WebPreferences/TLASE.jpg); background-size: cover; font-size:120%">
---+!! Database Growth - Strategies for minimizing the growth of repository databases  
%DKGRAY% Authors: Main.RosaNaranjo, Main.AlaEddineBannour, Main.TimFeeney,  Main.PaulEllis, Main.DanielMoul, Main.IngoUnruh </br>
Build basis: None
%ENDCOLOR%</div></sticky>

<!-- Page contents top of page on right hand side in box -->
<sticky><div style="float:right; border-width:1px; border-style:solid; border-color:#DFDFDF; background-color:#F6F6F6; margin:0 0 15px 15px; padding: 0 15px 0 15px;">
%TOC{title="Page contents"}%
</div></sticky>

<sticky><div style="margin:15px;"></sticky>

In planning or maintaining your IBM Engineering deployment, you may ask yourself these questions: 
   * "How do I ensure that my database size will not grow in an unmanaged way?"
   * "Are there any strategies I can put in place to help with this in the beginning?"
   * "Are there objects I should avoid storing in my repository?". 
There are some recommendations that will be captured in this page as well as some links to content that will help you understand.
</br>
We can break down those strategies into two approaches: approaches that you can apply to your current environment, and approaches that you should additionally consider when starting a new project or planning to deploy a new environment.
</br>
---++ Reduction Approaches
The following approaches are applied by each application if you want to reduce the data in your existing environment:<br />
 <img src="%ATTACHURLPATH%/reductionnolinks.png" alt="reductionnolinks.png" width="2724" height="1534" />


---+++ In DOORS Classic
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/doors/9.7.1?topic=folders-deleting-undeleting-purging-modules][Purging objects/modules]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/doors/9.7.1?topic=baselines-deleting][Deleting baselines]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/doors/9.7.1?topic=modules-deleting-objects][Deleting objects]]

---+++ DOORS Next
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.1.0?topic=areas-archiving-project-team][Archiving Project Areas]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/doors-next/7.1.0?topic=components-archiving-restoring][Archiving components]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.1.0?topic=applications-archiving-restoring-configurations][Archiving Configurations]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/doors-next/7.1.0?topic=artifacts-discarding-change-sets][Discarding of empty change sets]]. You can automate this approach with a script.
   * [[https://jazz.net/wiki/bin/view/Main/DNGDataPurge702][Data Redaction]]:  This approach is not intended for bulk deletion usage, and is only applicable for the redaction of sensitive data.  This is unlikely to reduce the database size.

The "safest" way to reduce database size, prior to the 7.x release, was to archive projects and use the repotools -deleteJFSResources command.  See the [[http://www.ibm.com/support/docview.wss?uid=swg21458233][Deleting data permanently from a DOORS Next Generation project]] technote if this is required as part of an upgrade from a release prior to 7.x.

---+++ Engineering Workflow Management/Rational Team Concert
In Engineering Workflow Management, formerly Rational Team Concert, there are several items that can contribute greatly to database growth if not kept in check.  These items are build results, workitem attachments, and binary content in SCM as versioned content. There are several articles and presentations that have been written on this topic.  

   * [[https://jazz.net/library/article//98165][Deleting File Versions in EWM: What It Really Means and How to Do It Safely?]]
   * [[https://jazz.net/wiki/bin/view/Main/ScmComponentDelete][Deleting SCM components and all their histories]]
   * [[https://jazz.net/wiki/bin/view/Deployment/MovingAComponentToANewServer][Moving SCM components to a new server]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/workflow-management/7.1.0?topic=tasks-deletebuildresults][Deleting the builds results]]
   * [[https://jazz.net/library/article/90403][Storing the build outputs in an external repository]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.1.0?topic=areas-archiving-project-team][Archving project areas]]

<br />Additionally, there are several articles and presentations that have been written on this topic: 
   * [[MovingAComponentToANewServer][Moving An Source Code Management Component To A New Server]]
   * [[https://trfeeney.wordpress.com/2014/12/09/help-my-rtc-database-is-getting-big/][Help! My RTC Database is getting big!]]
   * [[https://trfeeney.wordpress.com/2019/10/08/follow-up-to-help-my-rtc-database-is-getting-big/][Follow up to Help! My RTC Database is getting big!]]
   * [[DataDeletionRTC][Deleting data in Engineering Workflow Management]]
   * [[https://jazz.net/library/article/1459][Reducing the size of the Rational Team Concert repository database]]
   * [[https://jazz.net/library/article/1006][Deleting content in RTC]]

---+++ Engineering Test Management/Rational Quality Manager
For QM, there are some usages that can drive repository db growth.  For example, automation generating Execution Results and attachments on the test artifacts such as HTML, images, etc. attached to either scripts or results.  QM does have some ability to permanently delete content. See [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/test-management/7.1.0?topic=artifacts-deleting-restoring-test][Deletion and restoration of test artifacts]]. </br>
Other useful guides for Engineering Test Management are:

   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.1.0?topic=areas-archiving-project-team][Archiving project areas]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.0.3?topic=SSYMRC_7.0.3/com.ibm.jazz.vvc.doc/topics/t_comp_arch_rest_qm.htm][Archiving components]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.1.0?topic=applications-archiving-restoring-configurations][Archiving and Restoring Configurations]]
   * Deleting test cases results in bulk by creating a script using the [[https://jazz.net/wiki/bin/view/Main/RqmApi][API for deleting test cases results]]
   * Deleting test cases execution results by creating a script using the [[https://jazz.net/wiki/bin/view/Main/RqmApi][API for deleting test cases execution results]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/test-management/7.1.0?topic=artifacts-managing-artifact-attachments][Deleting artifact attachments]]

---+++ Systems Rhapsody Design:
 [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/rhapsody-model-manager/7.1.0?topic=administering-remove-sensitive-data][Removing diagram images from the server]]
<br />
---+++ In Lifecycle Query Engine (LQE):
[[https://jazz.net/wiki/bin/view/Deployment/LQERSDatabaseSizing][Moving to LQE relational store]] will need significantly less hardware and resources ( in RAM /Storage)  
<br />

---+++Data Warehouse
[[http://www-01.ibm.com/support/docview.wss?uid=swg21986358][Data warehouse database size is huge and growing fast]]

---+++ Lifecycle Index (LDX):
   * [[https://jazz.net/wiki/bin/view/Deployment/LDXrs710Performance][Moving to  LDX relational store]]
   * [[https://jazz.net/wiki/bin/view/Deployment/LDXrs710Performance][Decommissioning LDX and using only LQE Rs in v.7.1]]

All of the approaches that include archiving Data will reduce a the  Data that LQE is storing in its Index .
</br>
---++ Data Management
In addition to the above approaches, there are other approaches that you can implement when setting up a new project to ensure data efficiency, and that only the useful data is stored in the system. These approaches are again broken down per application: <br />
     <img src="%ATTACHURLPATH%/datamanagementnolinks2.jpg" alt="datamanagementnolinks2.jpg" width="2582" height="1512" />

---+++ DOORS Classic
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/doors/9.7.1?topic=baselines-deleting][Automated deleting of baseline  using a DXL script]]

---+++DOORS Next:
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.1.0?topic=areas-archiving-project-team][Archiving project areas and team areas]]
   * [[https://jazz.net/pub/new-noteworthy/erm/7.1/M7/index.html][Making a baseline in the past]]
   * Administrative setting to block uploading attachments above a defined size

---+++Engineering Test Management: 
   * Automated deleting test cases results in bulk by creating a script using the [[https://jazz.net/wiki/bin/view/Main/RqmApi][API for deleting test cases results]]
   * Automated deleting test cases execution results by creating a script using the [[https://jazz.net/wiki/bin/view/Main/RqmApi][API for deleting test cases execution results]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/test-management/7.1.0?topic=etm-setting-maximum-attachment-size][Blocking uploading attachments above the defined size]]

---+++Engineering Workflow Management/Rational Team Concert
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/workflow-management/7.1.0?topic=builds-automatically-deleting-build-results][Automatically deleting the builds results automatically]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.1.0?topic=server-enabling-external-content-repository-source-control][Storing the build outputs in external repository]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/workflow-management/7.1.0?topic=areas-enabling-external-attachment-storage-work-items][Storing work item attachments in an external storage]]
   * [[https://www.ibm.com/support/pages/size-limit-attachments-work-items][Blocking uploading attachments above a defined size]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/workflow-management/7.1.0?topic=lbruewmr-creating-reports-from-trs-feeds-scm-configuration-resources][Publishing/unpublishing configurations to LQE]]
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/lifecycle-management/7.1.0?topic=reference-scmdeletecomponent][Deleting SCM components using a repository tools command]]

---+++In Systems Rhapsody Design
   * [[https://www.ibm.com/docs/en/engineering-lifecycle-management-suite/workflow-management/7.1.0?topic=lbruewmr-creating-reports-from-trs-feeds-scm-configuration-resources][Publishing /unpublishing configurations to LQE]]


---++ Jazz.net forum resources
The forum contains many good suggestions from ELM Administrators who are maintaining ELM deployments. 

*Question:* I want to store binaries in EWM. Does EWM-SCM support delta compression?

*Answer:* Yes.  EWM-SCM supports delta compression of files, including binary files. In fact, one of the properties you can change in advanced properties is whether to enable or disable delta compression. If delta compression results in larger gains than other compression types it will be used to compress a given binary.

*Question:*  Can I prevent binaries from being checked-in to EWM?

*Answer:* Yes, you can implement file limits in EWM.

*Question:* Does EWM store all versioned elements in the database?

*Answer:* Yes, EWM-SCM stores all versioned elements in the database.

*Question* :What can I do to monitor db growth and what does EWM do on its own?

*Answer:*
   * EWM will run item cleanups daily (sometimes frequently throughout the day) to help clean stale data. You can investigate increasing this task, or the expiration time of these items (located in Advanced Properties)
   * You can also run the [[https://jazz.net/wiki/bin/view/Main/L3DevTool][onlineverify]] to ensure the consistency of the data and that there is no corruption which may be preventing the item clean up from running. 
   * Ensure all your builds are being pruned and any results cleaned up

*Question* How can you scrub !ChangeEvents from the repository database using IBM Engineering Workflow Management?

*Answer:* [[http://www.ibm.com/support/docview.wss?rs=3488&uid=swg21390701][Engineering Workflow Management repository database ChangeEvents scrubbing]]

Additional posts of interest:
[[https://jazz.net/forum/questions/92505/data-warehouse-db-sizing][Data warehouse db sizing]]

[[https://jazz.net/forum/questions/20098/are-there-ant-way-for-me-to-reduce-size-of-the-rtc-db][Are there any ways for me to reduce the size of the EWM db]]

[[https://jazz.net/forum/questions/73442/rtc-compression-for-binary-files-vs-clearcase][RTC compression for binary files vs clearcase]]

[[https://jazz.net/forum/questions/66014/rtc-large-history-complexity-and-archiving-questions][RTC large history complexity and archiving questions]]

[[https://jazz.net/forum/questions/128039/what-is-the-latest-metrics-by-namespace-report-means][What is the Latest Metrics by Namespace report?]]

[[https://jazz.net/forum/questions/19251/rather-surprising-db2-growth][Rather surprising db2 growth]]

[[https://jazz.net/forum/questions/61330/expected-database-growth-when-versioning-binary-data][Expected database growth when versioning binary data]]

[[https://jazz.net/forum/questions/65323/versioning-binary-artifacts#page=0&type=&q=database%2Bgrowth][Versioning binary artifacts]]


---+++++!! Related topics: [[DeploymentPlanningAndDesign][Deployment Planning and Design]], [[WhyIsMyDatabaseConsumingSoManyResources][Why is my database consuming so many resources?]]

---+++++!! Additional contributors:  StefVanDijk, BenjaminSilverman, KrzysztofKazmierczyk

<sticky></div></sticky>
Attachments
Topic attachments
I	Attachment	Action	Size	Date	Who	Comment
gif	RDNG.gif	manage	18.2 K	2016-07-07 - 17:21	RosaNaranjo
png	datamanagement.png	manage	146.8 K	2025-09-12 - 17:39	PaulEllis	Data Management
jpg	datamanagementnolinks2.jpg	manage	442.4 K	2025-09-15 - 06:02	PaulEllis	Data Management image
png	reductionnolinks.png	manage	452.1 K	2025-09-15 - 05:49	PaulEllis
Copyright © by IBM and non-IBM contributing authors. All material on this collaboration platform is the property of the contributing authors.
Contributions are governed by our Terms of Use. Please read the following disclaimer.
Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.