Does running Data Warehouse Collection Jobs multiple times a day affect data output?
Hi people,
We have some reports that consumes data from data warehouse, and we need this reports to be updated more than once a day. Currently, the data warehouse collection jobs (ETLs) are scheduled to be run once a day. If we schedule the jobs to be run more than once a day, could this affect some reports? For example burndown charts?
I will really appreciate your help.
Regards,
2 answers
How frequently the Work Items get updated for you? If the rate is high then the frequently running the data collection jobs will make the reports refresh faster.
However, note that most data collection jobs do a delta fetch. Thus, if the update rate is low, then they will anyway not be fetching anything and you wont see any difference in the reports.
Running data collection jobs is not cheap, specially if the number of items in the system is high. Choose a schedule such that you don't put unnecessary pressure on the servers. It will be a trade off between how latest you want your reports to be and how much load you wish to put on the servers.
However, note that most data collection jobs do a delta fetch. Thus, if the update rate is low, then they will anyway not be fetching anything and you wont see any difference in the reports.
Running data collection jobs is not cheap, specially if the number of items in the system is high. Choose a schedule such that you don't put unnecessary pressure on the servers. It will be a trade off between how latest you want your reports to be and how much load you wish to put on the servers.
The Burndown report will always show a last live point. So it really does not need to run the ETLs more frequently.
Most of the ETL jobs are delta. But the trending historical data is not delta and can consume a large amount of disk space.
So it is really not recommended to run the bobs more than once a day. In 4.0, we actually removed to option to run the ETLs more frequently.