It's all about the answers!

Ask a question

Optimal DCC scheduling with Data Manager jobs

June Boston (1942438) | asked Oct 20 '15, 12:42 p.m.

I saw a recommendation on these forums that, in a purely DCC strategy, the ODS Data Collection jobs can be scheduled to run frequently throughout the day but the Data-Mart Data Collection jobs only need to be run once a day and gain no benefit from frequent runs.  Presumably this is why they have separate schedulers.

When running DCC jobs from Data Manager, it's just a black box where you run the job to trigger the dcswrapper java class, which connects to DCC and executes whatever jobs you have turned on.  There is no ability to select specific jobs, it is all or nothing.

My question is: could we use the DCC scheduler -and- Data Manager, running the DCC ODS jobs every few hours or so, the Data-Mart jobs in the evening, and our jobs at night at a time that will fit between the ODS jobs?  Our jobs take a half hour to run.  You cannot really tell the ODS jobs to only run between certain hours, but there should be so few changes around midnight that if we set them to run every 4 hours and ran our custom jobs 2 hours after they start, we should fit in the gap safely.

Our custom ETL jobs get additional data from CLM, do some calculated fields, and make custom attributes easier to access with Rational Insight.  They query from the existing dw tables but they write to their own tables.  Maybe there would be problems if they queried a table at the same time as that table was being written to by DCC.  Could we test that being problematic?  Is it safe for two processes to request things from one table on a multi user SQL database?  If DM and DCC never run at the same time, will we be ok?

One answer

permanent link
Rafik Jaouani (5.0k16) | answered Oct 20 '15, 1:01 p.m.
Hi June, the DCC and DM ETLs should never be run at the same time. But I guess you could set up both the DCC and DM schedules to run at times that are far away from each others. Another solution which I have not tried but i do not see why it should not work. You could configure two DCC instances. One that runs the ODS jobs and another that runs the Datamart jobs and invoke both instances from data manager at different times.

Your answer

Register or to post your answer.