Jazz Forum Welcome to the Jazz Community Forum Connect and collaborate with IBM Engineering experts and users

CCM export process as part of DB migration - badly defragments the disk. Any root cause known?

Questions
Tags
Users
Badges
Deep Bhowmik
(11) Apr 24 '23, 4:12 a.m.

 We are preparing for our upgrade from SQL Server to DB2 and from 6061 to 702. We are using the "repotools-xxx -exportConcurrent" and "repotools-xxx -importConcurrent" commands to transfer the data from the existing SQL Server databases to the new DB2 databases.

We have run thru the process a number of times in our QA/Staging environment. GC, JTS, RM, QM, and DCC all work perfectly. We have however run into an intermittent error with exporting our biggest database - i.e. CCM ( ~ 1 TB )

At 80% (about 7 hours into the CCM exportConcurrent command), the job raises an exception stating that a "File System Limitation" has occurred. We checked and before running the command, the disk drive was 0% fragmented. However, after the command failed, it is 83% fragmented. This has happened in the past and we sometimes succeeded in defragmenting completely or partially, but not always. After a few tries, the disk still remains badly defragmented.


We have also replaced the disk and tried with a brand new one. The issue still occurs. Looks like the export process, especially for a large DB, is causing the target disk to badly defragment. The server where the disk is mounted, runs on Windows2016.

Does anyone have any experience on this issue?


0 votes

2 answers
3,441 views
0 votes

2 answers

Permanent link
Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER (64.6k●3●36●49) Apr 24 '23, 4:37 a.m.

 I would suggest to open a case with support. They can get the logs and details about the error message.


" is causing the target disk to badly defragment"

Being defragmented is a good thing - being fragmentated is slightly worse, but should also not be an issue here. See https://en.wikipedia.org/wiki/Defragmentation for what this actually means.

As far as I can tell, the fragmentation of the file system is nothing that should be of any interest or impact. Any disk that is used with random read/write operations will have a certain fragmentation. This can make read/write operations a tiny bit slower, because not all blocks of a file are in sequence on disk, but should have no other impact. You do not have to replace drives just because of fragmentation.

0 votes

Comments
Deep Bhowmik
Apr 24 '23, 4:54 a.m.
Thank you Ralph for your answer. We did open a case with support and awaiting a feedback.   One clarification first -"is causing the target disk to badly defragment" had a typo in there. It would read " is causing the target disk to badly fragment". Well the issue here is not that fragmentation is our main worry. But that the export command fails after completing ~90%, throwing a file system limitation error. At that point we find that the disk is badly fragmented. Last time, we had defragmented and ran the process again, it failed, and again, then it succeeded i.e. completed export successfully.  This week we are running the export again and the same issues are coming. The export fails.

Ralph Schoon
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Apr 24 '23, 5:02 a.m.
It is necessary to work with support to find out what the root cause of the file system limitation error is. I am pretty sure that the fragmentation is of no concern. From my experience with disk fragmentation this should not cause anything. If you would use an SSD for the drive it would even not consider fragmentation a thing.    

Deep Bhowmik
Apr 24 '23, 5:32 a.m.
Thank you for sharing your experience and thought. It gave me another perspective.  We know for a fact that A. CCM export is failing, and B. Disk fragmentation is occurring. And then we infer that A is caused by B. Now that cause-effect relation may be our assumption, and these two incidences might be independent of each other.  

David Honey
FORUM ADMINISTRATOR / FORUM MODERATOR / JAZZ DEVELOPER Apr 24 '23, 5:50 a.m.
On SSDs, fragmentation is largely irrelevant. On hard disks backed up by SSD caches (such as Optane) or large memory caches, the effects of fragmentation are usually mild. Fragmentation is just a fact of life. Almost anything you do that creates large files that span free block sizes will cause fragmentation. On hard disks without large SSD/memory caches, defragmenting the disk should be regular housekeeping. Once a month seems like the least often you should do this.

Permanent link
Georg Kellner
(840●5●82●110) Apr 25 '23, 8:37 a.m.

 Do you use the external storage option for the SCM data?
Might be using this option would make things easier.


So you can activate/use this feature to store files bigger than 1 MB on local storage, start the database export from a database which should be lots of GB smaller.
After the db migration you can use the external storage or you can revert the changes.

greetings georg.

0 votes

Your answer

Register or log in to post your answer.

Dashboards and work items are no longer publicly available, so some links may be invalid. We now provide similar information through other means. Learn more here.

Ask a question
Guidelines
FAQ
Search context
Follow this question

By Email: 

Once you sign in you will be able to subscribe for any updates here.

By RSS:

Answers
Answers and Comments
Question details
× 7,622
× 1,452
× 120
× 75

Question asked: Apr 24 '23, 4:12 a.m.

Question was seen: 3,441 times

Last updated: Apr 25 '23, 8:37 a.m.

Confirmation Cancel Confirm