Missing data updates
Over the months we have had performance issues, and have killed tasks along the way or had the RTC/RQM application server drop for various reasons. based on the identified invalid data it does appear to occur in similar time frames. Two Questions:
- What is the best way to ensure the Data Warehouse is current with out rebuilding the whole data warehouse from scratch? We have been making individual changes to the master application data (adding a tag for example) and this seems to fix that one. We are thinking of resetting the time to February and running the ODS/DM tasks.
- If we have to kill tasks in the future, what is the recommended method to recover? He have adopted setting the date to today (or the day of the failure) and letting ODS/DM run.
One answer
If your DW is not up to date, but you don't want to rebuild it entirely, then setting up a specific date for the delta ETL would be the right work around. Keep in mind though, that the data mart data does not get reprocessed. If you run the data mart jobs today, all the totals would be stored as today's totals. If you run it again tomorrow, after setting a previous date for the delta ETL, the totals would be stored as tomorrow's totals.
I think a more important question would be, why do you have to kill data collection tasks? I would recommend that you contact IBM Support so they can help you look into the issue and avoid future data loss.
Thank you,
Clara Forero
Comments
Thanks - What is the impact of the data being stored for various days totals? Could a change from a week ago appear as if it occurred today?
We had to kill some jobs which occasionally decide to take hours to run instead of minutes... It appears to be a defect in the product. We've had a PMR open for month but progress seems to be very slow. (we provided wait data while things were slow awhile ago (3 weeks?)... but we have been battling the DCC tasks for months. Users are noticing incorrect information and we need to find a way to get it all corrected.
Is there any documentation/information about how this all works? It is a black box to use now and difficult to know how to address issues we encounter.
Hi Gene,
I'm not sure what you mean by "What is the impact of the data being stored for various days totals?" If it refers to the comment I made about the data mart ETL's, what I meant with that was that for the metrics tables you will have missing days. Every time the data mart ETL's get executed, the records created up to that point get summarized and stored in the metrics tables as the totals for that day. If the data mart ETL's don't run on a specific day, then it won't be any totals calculated for that day and if you are using the metrics tables for reporting, you'll see missing days in them.
This link contains information about Reporting in CLM, including a link to the Data Collection Component (DCC) documentation.
Thank you,
Clara